OMG the server is dead

It is the scenario every homelabber dreads: a catastrophic failure. My HP MicroServer Gen8 didn’t just lose a drive; the primary RAID 1 array holding the OS suffered a double-disk death.

Starting from scratch with Fedora 43, I had two goals: build a bulletproof, redundant boot drive on new hardware, and recover my massive 4-disk RAID 10 array that survived the crash.

It wasn’t straightforward. The Gen8 is aging hardware (Legacy BIOS) trying to run bleeding-edge software. Here is how I fought the installer—and won—and how I tricked LVM into finding my old data.

Part 1: The OS Installation (Legacy BIOS vs. Modern Storage)

The first hurdle was the HP MicroServer Gen8 itself. It predates the modern UEFI standard, meaning I couldn’t use the standard EFI partition layout that Fedora defaults to. I had to use the Blivet-GUI (Advanced Custom) installer to manually architect a layout that satisfied both the old BIOS and modern redundancy needs.

The Partition Strategy

To ensure that next time a drive failure doesn’t kill the system, I used a hybrid approach:

1. BIOS Boot Partition (1 MiB): Created on both disks (sda and sdb). This is required for GRUB to embed itself on GPT disks in non-UEFI systems.
2. /boot (1 GiB, ext4, MDRAID 1): I stuck to standard Linux Software RAID here. GRUB hates booting from Btrfs RAID 0/1, so a standard mirror for the kernel is safer.
3. / Root (Btrfs RAID 1): For the OS and data, I used Btrfs Native RAID 1. This gives me data mirroring and bit-rot protection (self-healing).

The “Day 2” Bootloader Fix

The installer successfully set up the RAID, but it only wrote the bootloader to the first disk. If sda died, sdb wouldn’t have known how to boot.

The Fix:

I had to manually install GRUB to the second drive’s MBR immediately after the first boot:

sudo grub2-install /dev/sdb

Now, if a drive fails, the BIOS simply hops to the next disk, and the system stays up.

Part 2: Recovering the “Invisible” RAID 10

With the OS running, I needed to re-attach my main storage: a 6TB RAID 10 array sitting on an LSI controller.

Linux saw the raw device immediately as /dev/md126 (via lsblk), but I couldn’t access the data.

The Error: LVM2_member

Trying to mount the device directly failed with “unknown filesystem type ‘LVM2_member’.”

This meant the array wasn’t a file system; it was a physical container for LVM (Logical Volume Manager). However, running sudo vgs (Volume Group Scan) returned… silence. LVM refused to admit the drive existed.

The Blocker: The “Devices File”

Fedora has introduced a strict LVM Devices File. It essentially whitelists which disks LVM is allowed to look at. Since my old RAID array wasn’t in the new OS’s “allow list,” Fedora was actively ignoring it to prevent conflicts.

The Solution:

I had to force-add the RAID device to the LVM whitelist:

sudo lvmdevices --adddev /dev/md126

Once added, I refreshed the scan:

sudo vgscan
sudo vgchange -ay

Suddenly, the volume group lvm-raid appeared!

The Final Mount

With the volume group active, I found the logical volume name (Raid) using sudo lvs and mounted it properly:

sudo mount /dev/lvm-raid/Raid /media/storage

Summary

The HP MicroServer Gen8 is still a beast, but it requires care when pairing it with modern Linux:

1. Avoid EFI partitions; use “BIOS Boot” on both drives.
2. Manually install GRUB to your secondary drive for true failover.
3. Watch out for LVM Filters: If your old RAID array won’t show up, check lvmdevices—it’s likely being blocked by safety features.

The server is now back online, fully redundant, and running Fedora 43.