I bought a Raspberry Pi 5 with an NVMe base from Pimoroni and decided to play around with it. I should mention that it has been quite a while since my last Raspberry Pi endeavours. 4 years to be exact, so I’m a little rusty.

As I hadn’t bought or owned a micro HDMI cable, everything had to be done headless and over the network¹. Additionally, I also couldn’t find my microSD cards anywhere, so no main storage, no display output. Just USB, NVMe and a network cable.

Chapter 1: Fedora IoT

As a main Fedorian, using Raspberry Pi OS, sounded not only boring, but also a bit annoying, as I’m not a particular fan of the Debian package toolchain. And immutable OSes, like Fedora IoT, have some other advantages as well. I decided to give Fedora IoT a try. Flashed it onto a USB stick, plugged it into the Pi, power, Ethernet, let’s go.

And nothing. The Pi never got an IP, the cooler never slowed down, I’m sure it didn’t boot. A little bit of research later: Raspberry Pi 5 and Fedora IoT, currently not a thing, the kernel support isn’t there yet.

Fixing the kernel problems myself, wasn’t an option, so Raspberry Pi OS it is.

Chapter 2: Raspberry Pi OS via USB

Getting Raspberry Pi OS up and running is pretty simple, download the ISO, verify it, put it on the USB stick with dd, sync, eject it safely, plug in the Raspberry Pi, boot it, let’s go.

Results: It boots, it gets an IP and you can’t SSH into it. Raspberry Pi OS doesn’t enable SSH by default (safe default, but not helpful in my situation). So I turned off the Pi, plugged the USB stick back into my regular notebook and created a ssh file in the boot partition.

Once it was in the Raspberry Pi, I was able to ping it and even open an SSH connection, but I couldn’t log in as they no longer do the bad thing with default passwords that people never change. So once again, unplug, go to the PC, create a userconf.txt file in the boot partition, put it back on the Pi, it boots, SSH works. Nice!

Chapter 3: Raspberry Pi OS via NVMe

With everything up and running via USB, it was time to use the NVMe. Knowing that Raspberry Pi OS would work and not really knowing why Fedora IoT wouldn’t, I decided to flash Fedora IoT to the NVMe using a dd command in Raspberry Pi OS. Rebooted and… it booted back to Raspberry Pi OS. Apparently Fedora IoT wasn’t booting properly from the NVMe either, so the bootloader switched back to the USB stick with Raspberry Pi OS.

Time to flash Raspberry Pi OS using the same method. Successfully flashed, reboot and the Pi comes up, chooses an IP. …but no SSH.

Oops, I forgot to create the ssh file and userconf.txt. But since the boot order favoured the NVMe over the USB drive, there was no way to boot from USB. I could access it using a keyboard and micro HDMI, but I don’t have the cable for that.

But hey, I’m a smart guy, so I’ll just unplug the NVMe, boot from USB, plug it back in and create the files. So I unscrew a couple of screws, remove the NVMe, boot the thing up and plug the NVMe back in, with one hand holding the Raspberry Pi up so I don’t damage the PCIe ribbon cable. And… nothing. The NVMe doesn’t appear. No hot plugging.

Even triggering a manual rescan, didn’t work. Time to ask the Internet for help. First suggestion: Change the boot order with raspi-config, great idea! Unplug the NVMe, boot from USB, run raspi-config, open the boot order settings: Whoops, no boot order option that prefers USB over NVMe. Well, nice try.

Chapter 4: Hacking the boot order

But there was more, w4tsn, who uses Raspberry Pis extensively, suggested: Why not modify the boot order in the EEPROM directly? And delivered a ready-to-use boot order code 0x456f.

With the Pi still booted, I went ahead. One sudo -E rpi-eeprom-config --edit later, I replaced the boot order configuration, with the suggested one, thinking: “This looks a bit strange, very different from the default configuration.”, and then rebooting.

It didn’t come back. Reading the documentation again I noticed the very important sentence: “It is read right to left, […].” Whoops. That’s why all default configurations start with 0xf, like 0xf41. Lesson learned: Always double-check, even experts might miss small details on their day off.

Chapter 5: Bringing it all back to life

Bricking a Raspberry Pi using the boot order is certainly a first for me. But with some time to reconsider and some more helpful people chipping in, I was able to find rpiboot. rpiboot is a tool for recovering the Raspberry Pi bootloader or flashing the eMMC of your Raspberry Pi Compute model.

In my case, the first step was to recover the EEPROM. Since I’m running Fedora Silverblue, compiling it natively wasn’t an option², so I wrote a Dockerfile that worked somewhat, at least for boot phase 1. However, as the Raspberry Pi has two boot phases where rpiboot has to re-detect the device, this failed in unexpected ways.

Finally I wrote a second Dockerfile for Fedora 40, in sync with the Fedora Silverblue 40, that I’m running, compiled rpiboot and ran it natively. From there it just worked like a charm.

Recovery booted, EEPROM was reset, now the Raspberry Pi could boot from USB again. The NVMe was unplugged at this point.

Chapter 6: The intended setup

Using my newfound ability to use rpiboot, I used the mass-storage-gadget64 directory to boot the Raspberry Pi in a mode where it appeared as a mass storage device on my notebook, and I could just flash the image directly to the NVMe. No USB sticks, no fuss.

I even used the official Raspberry Pi Imager Flatpak, selected the NVMe and flashed the OS, with SSH enabled. I also deliberately increased the OS partition to 100GB using GNOME Disks and added an encrypted LUKS partition to store data.

Now all I need is a Network Bound Disk Encryption setup to automatically unlock the LUKS partition and I’m finally ready to use the Raspberry Pi.

Conclusion

This has been a much wilder ride than I expected, and there were several times when I thought “well, this is messed up, I hope there is a way to undo this”. And it seems that the people designing the Raspberry Pis have a lot of these types of customers and have managed to make it quite fail-safe.

I learned more than I expected and it was an interesting challenge. But there is also some Raspberry Pi documentation that really needs some polishing.