Fedora Local Server Troubleshooting Notes and Deferred Work

Not Everything That Looks Broken Is Broken

A clean guide can hide an important reality: several things that look wrong during a local-server build are expected behavior, while some easy-looking fixes would weaken the final system.

This reference collects the implementation lessons that generalize well enough to help another reader without forcing them through the original debugging transcript.

Example values: Hostnames, usernames, addresses, interface names, device paths, and other identifiers are illustrative unless explicitly described as software defaults. Discover the values on your own systems before running commands.

Diagnose from your own virsh, ip, lsblk, tailscale, sshd, systemd, and journal output before applying a fix. The entries below describe patterns from this build, not a promise that another machine has the same cause.

A Virtual Bridge Can Be Healthy Without Carrier

ip -br address show virbr0 shows the bridge as DOWN, or ip link show virbr0 reports NO-CARRIER, before any VM is running. That can be normal: a libvirt bridge can exist before any guest tap interface is attached.

Inspect the libvirt network first:

sudo virsh -c qemu:///system net-list --all
sudo virsh -c qemu:///system net-dumpxml default

Libvirt's documented default NAT network creates a host bridge such as virbr0 and dynamically connects guest-side tap devices such as vnetN when domains attach.¹

After starting a VM, re-check the host-side links:

ip -br address show virbr0
ip -br link | grep -E 'virbr|vnet'

Fedora users have repeatedly encountered bridge states that look suspicious before a guest is attached. Community reports are useful context, but the authoritative question is whether libvirt says the network is active and whether the guest actually attaches and routes correctly.²

A carrier-less virbr0 is not a reason to rebuild the network as a LAN bridge.

A Missing Legacy Service Does Not Mean Virtualization Is Broken

libvirtd.service is missing or inactive, and an older tutorial treats that as a broken virtualization stack. Modern libvirt supports modular daemons such as virtqemud instead of requiring one monolithic libvirtd process.³

Test the actual system connection:

sudo virsh -c qemu:///system list --all
systemctl status virtqemud.socket --no-pager

If qemu:///system works, the virtualization control plane may be healthy even when an older tutorial's service-name assumption does not match your Fedora release.

A Virtual Network Does Not Create a Storage Pool

The default NAT network and a storage pool named default are separate libvirt objects. A working virtual network proves nothing about storage-pool existence.

Inspect first:

sudo virsh -c qemu:///system pool-list --all

If you intentionally need a directory pool at /var/lib/libvirt/images:

sudo mkdir -p /var/lib/libvirt/images
sudo restorecon -Rv /var/lib/libvirt/images
sudo virsh -c qemu:///system pool-define-as default dir --target /var/lib/libvirt/images
sudo virsh -c qemu:///system pool-start default
sudo virsh -c qemu:///system pool-autostart default

Libvirt's storage model treats pools and volumes as their own managed objects.⁴

A Temporary Storage Pool May Be Registration, Not Duplication

virt-manager can register a user directory such as ~/Downloads when you select installation media from it.

Inspect before touching it:

sudo virsh -c qemu:///system pool-info Downloads
sudo virsh -c qemu:///system vol-list Downloads
sudo virsh -c qemu:///system pool-dumpxml Downloads

If the pool only registers an ordinary directory and you want libvirt to stop managing it:

sudo virsh -c qemu:///system pool-destroy Downloads
sudo virsh -c qemu:///system pool-undefine Downloads

A pool definition and the files beneath its backend are related, but they are not interchangeable concepts.^4:1

The Workstation Is Not Supposed to See the Guest NAT Address Directly

With libvirt's default NAT model, the workstation usually has no route to the guest's private subnet. The default network is NATed by design, so guest addresses are not ordinary LAN peers.^1:1

admin workstation
       │
       └── local LAN
             │
             └── physical Fedora host
                   │
                   └── libvirt NAT subnet
                         └── VM

So this may fail from the workstation:

ssh admin@<GUEST_LIBVIRT_IP>

while the physical host can reach that same guest address.

Use ProxyJump through the host as a LAN fallback, or use the VM's own Tailscale identity for normal administration. Do not expose the private libvirt subnet to the physical LAN just to avoid the NAT boundary.

ProxyJump Does Not Hand the Host Your Private Key

Suppose the VM's administrative private key exists only on the admin workstation.

When you run:

ssh server-vm-01-lan

OpenSSH on the workstation authenticates to the VM. The physical host forwards the TCP stream; OpenSSH does not delegate the workstation's private keys to the jump host.⁵

If you first log into the physical host and then run:

ssh admin@<GUEST_LIBVIRT_IP>

a different SSH client is now running on the host. If the host does not possess an authorized VM private key, the guest should reject it.

Do not copy the workstation's VM private key onto the host just to make that second command succeed. If host-to-VM automation becomes a real requirement, create a separate purpose-specific identity with scoped authorization.

ProxyJump Failures Need Layered Diagnosis

When ordinary host-to-guest networking works but ProxyJump does not, separate the questions:

1. Can the physical host reach the VM IP?
2. Can it establish TCP 22 to the guest?
3. Does the guest sshd listen and accept the intended identity?
4. Does the physical host's effective sshd policy permit forwarding?
5. Is SELinux denying the forwarded connection?

Check the forwarding policy and recent SELinux evidence:

sudo sshd -T | grep -E 'allowtcpforwarding|disableforwarding|permitopen'
sudo ausearch -m AVC -ts recent
sudo journalctl -t setroubleshoot --since "10 minutes ago"

Fedora's SELinux troubleshooting guidance recommends correcting the narrow policy or configuration cause instead of disabling SELinux globally.⁶

If you install a local SELinux policy module for a confirmed gap, document it and revisit whether later Fedora policy updates make it unnecessary.

One Server Can Have Several Addresses and One SSH Identity

The first SSH connection to a Tailnet address may report that the address is new while also noting that the same host key is known under another address.

Inspect the server's ED25519 fingerprint locally:

sudo ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub

If the fingerprint matches the trusted LAN identity, accepting the same server under its Tailnet address can be reasonable.

If the fingerprint is different and you did not deliberately reinstall or rotate host keys, stop and investigate. Host-key verification is one of OpenSSH's core server-authentication controls, not a warning to suppress reflexively.^5:1

A Tailscale CLI Is Not the Same Thing as a Running macOS Client

tailscale still runs in the shell, but the Tailscale app or daemon is not running. On macOS, the CLI and the active network service are not always the same component. Tailscale documents multiple macOS variants: Standalone, App Store, and an open-source CLI/daemon variant, with different packaging and extension behavior.⁷

Check both application presence and shell resolution:

ls -ld /Applications/Tailscale.app
open -a Tailscale
type -a tailscale

If a CLI remains but the chosen app or daemon is absent, restore one supported installation model instead of mixing variants.

Version Mismatch Warnings Start with Binary Discovery

Find every CLI on the shell path and check for a Homebrew package:

type -a tailscale
brew list --versions tailscale 2>/dev/null || true

For the Standalone app, compare the bundled CLI behavior:

TAILSCALE_BE_CLI=1 /Applications/Tailscale.app/Contents/MacOS/Tailscale version

Tailscale's issue history contains both genuine split-install mismatch cases and false-positive version-string mismatches. "The versions differ" is a clue, not a diagnosis.⁸

Use the current macOS and CLI documentation to decide which binary belongs to the installation model you selected.^7:1

Reachability and Application Authorization Are Different Layers

A successful tailscale ping does not prove SSH or another service is permitted.

Use policy-relevant traffic and then test the actual application:

tailscale ping --icmp local-server
ssh local-server

If a peer is visible but SSH is denied, review the Tailnet Grant, host firewall, and OpenSSH layers before changing unrelated routing.

Tailscale Grants are deny-by-default once broad allow rules are removed, and Tailscale recommends policy tests as part of least-privilege administration.⁹

Key Expiry Can Turn a Network Path into an Authentication Problem

An expired Tailscale node must reauthenticate before participating normally again.

On a machine you control through an independent local path, reauthentication may use:

tailscale up --force-reauth

Do not run that casually over the only remote session that depends on the same Tailnet.

Tailscale documents trusted servers, subnet routers, and difficult-to-reach remote devices as cases where disabling key expiry can make operational sense.¹⁰ It is a role-based availability choice, not a requirement for every server.

Desktop-Mounted Storage Is a Sign the Ownership Model Is Wrong

If long-lived server data is being unlocked and mounted through a desktop session under:

/run/media/<user>/...

move the persistent configuration to system-level storage management:

/etc/crypttab
/etc/fstab
/srv/storage

GNOME's GVfs volume-monitor policy documents x-gvfs-hide and stable /dev/disk/... identifiers for controlling how persistent filesystems appear in the desktop UI.¹¹

Remember the two identities:

outer LUKS UUID       → crypttab
inner filesystem UUID → fstab

Missing Optional Storage Can Be a Healthy Boot

If both the encrypted mapping and mount use nofail, /srv/storage can be absent after boot without making the host unhealthy.

Check the block-device state first:

lsblk

If the mapper is absent, unlock the encrypted volume by the LUKS UUID recorded in Guide 05 and mount it:

sudo cryptsetup open /dev/disk/by-uuid/<SECONDARY_LUKS_UUID> server-storage
sudo mount /srv/storage

If something failed unexpectedly, inspect the boot journal:

journalctl -b | grep -Ei 'crypt|server-storage|srv-storage'

systemd's crypttab documentation explicitly states that nofail makes the encrypted device non-fatal but a dependent mount needs its own nofail too if boot should continue without the volume.¹²

A GDM Suspend Broadcast Does Not Prove the Server Slept

An SSH session can receive a wall message from gdm-greeter such as The system will suspend now! even after the systemd sleep targets were masked. GNOME has its own idle-power policy, and GDM runs a separate login-greeter session with its own dconf profile.¹⁷

First verify the server-side safety net:

systemctl is-enabled sleep.target suspend.target hibernate.target hybrid-sleep.target

For an always-on host using this series' baseline, all four targets should still report masked.

Then inspect the current boot around the event instead of inferring from the broadcast alone:

journalctl -b | grep -Ei 'suspend|sleep|gdm|logind'

systemd documents masking as making a unit impossible to activate normally, while systemctl suspend starts the suspend path through suspend.target.¹⁸ The broadcast therefore tells you that the graphical layer requested suspend; by itself it does not prove that the machine actually entered a sleep state.

If the server should never make the request either, configure the GNOME/GDM automatic-suspend policy deliberately rather than unmasking the systemd targets. GNOME exposes settings such as sleep-inactive-ac-timeout and sleep-inactive-ac-type, including a nothing action.^17:1 Keep the two layers conceptually separate: desktop policy controls the request, and the systemd mask remains the server-side backstop.

Cockpit Is Optional Even When Fedora Enables It

Fedora Server may have cockpit.socket enabled. If SSH is your intended administration surface and you do not want browser administration:

sudo systemctl disable --now cockpit.socket
systemctl is-enabled cockpit.socket
systemctl is-active cockpit.socket

Cockpit's own documentation says the web service starts on demand through cockpit.socket, normally on port 9090, and can be disabled if browser access is not wanted.¹³

In a no-Cockpit baseline, the socket should be disabled and inactive.

Firmware Capability Does Not Mean an Update Is Available

Use fwupdmgr get-upgrades to see actual available firmware updates:

fwupdmgr get-upgrades

The fwupdmgr reference defines get-upgrades/get-updates as the operation that shows real available updates.¹⁴

A device may be supported and update-capable while already running the latest firmware available through the enabled remotes.

An Encrypted Root Can Make a Healthy Server Look Offline After Reboot

If the Fedora root disk is passphrase-encrypted, a machine that never returns to SSH or Tailscale may simply be waiting at the LUKS unlock prompt.

That is an availability consequence of full-disk encryption, not an sshd or Tailscale failure. Those services cannot start until Fedora unlocks and mounts the root filesystem.

If unattended reboot becomes a requirement, evaluate supported mechanisms such as TPM2 or FIDO2 enrollment through systemd-cryptenroll. Treat that as a security-model change, not a free convenience feature.¹⁵

Deferred Work Stays Outside the Baseline

The remaining topics are deliberately outside this series. Each is real work, but each depends on a decision the baseline does not make: the final network edge, a module-signing policy, an unlock threat model, a secrets architecture, or the point at which the LAN fallback is no longer needed.

The Network Edge Belongs in Its Own Project

This collection stops at the local-server boundary. A separate network guide can cover dedicated firewall hardware, OPNsense, ISP-router bypass or bridge modes, managed switching, Wi-Fi access points, VLAN segmentation, DHCP reservations, inter-VLAN policy, and network observability.

Avoid baking temporary router assumptions deeply into the server configuration when you already know the physical network will change.

Tailscale is useful during that transition because routine server administration can remain stable while LAN addressing moves underneath it.

Secure Boot with Third-Party Modules Needs Its Own Test Plan

Secure Boot can strengthen boot-time trust, but Fedora's Secure Boot chain rejects kernel modules that are not trusted by the configured signing policy.¹⁶

If the host uses proprietary or locally built modules, treat signing and enrollment as a separate hardening project. Do not enable Secure Boot during unrelated cleanup and discover afterward that a required kernel module can no longer load.

Automatic Secondary-Disk Unlock Should Come After the Manual Model Is Clear

The baseline secondary-storage design asks for a passphrase. Future options include key files protected by an encrypted root filesystem, TPM2, FIDO2, PKCS#11 devices, and other systemd or cryptsetup-supported mechanisms.

systemd-cryptenroll documents several hardware-backed enrollment paths.^15:1

Each trades some manual recovery friction for a different trust dependency. Automate unlock only after deciding which threat you are trying to defend against.

Secrets and SSH Recovery Deserve a Separate Architecture

This series establishes only the minimum recovery rule: if SSH passwords are disabled, losing the only private SSH key can eliminate remote authentication even when the network path itself is healthy.

A dedicated recovery and secrets guide should cover:

• offline SSH recovery identities;
• password managers;
• hardware security keys;
• Tailnet Lock disablement secrets;
• API keys and service credentials;
• age/SOPS;
• Vault/OpenBao-style secret services;
• encrypted or offsite recovery media;
• recovery drills.

A basic server guide should not become the canonical place where secret material gets copied.

Remove ProxyJump Only After You No Longer Need the Independent Path

Once direct Tailscale access is stable, the LAN jump path may no longer be necessary for daily use. It can still be valuable during physical-network migration because it is an independent local recovery path.

After the final LAN is stable, reconsider:

• whether AllowTcpForwarding yes remains required;
• whether any local SELinux forwarding workaround can be removed;
• whether server-vm-01-lan is still useful.

Remove recovery paths because the design no longer needs them, not because the primary path has been convenient for a week.

FAQ

Keep the Boundary Intact While You Debug

The strongest lesson from this build is not one command. It is a troubleshooting posture:

Keep the security boundary intact, verify each layer independently, and make the smallest change that explains the observed failure.

That means:

• inspect routing before adding routes;
• inspect effective sshd policy before editing it;
• inspect SELinux denials before disabling SELinux;
• inspect libvirt objects before replacing network topology;
• inspect which executable is running before reinstalling packages;
• inspect disk identity before editing mount configuration;
• validate recovery paths before removing old ones.

That approach turns a repurposed desktop into infrastructure without turning every temporary problem into permanent complexity.