sharing preview • dzombak.com

Running a Raspberry Pi with a read-only root filesystem

While there are downsides, using a read-only root filesystem will dramatically increase your Pi SD card's lifetime.

Running a Raspberry Pi with a read-only root filesystem

March 29, 2024 • Tagged: series:pi-reliability raspberry-pi linux

Part of the Raspberry Pi Reliability series.

Many applications that run on Raspberry Pis and similar single-board computers — for example, environmental data loggers that report to a central database server — don’t really need to store any state locally on the Pi’s SD card. This means you can run the Pi with a read-only root filesystem, which will dramatically increase the SD card’s lifetime.

Keep in mind that, with a read-only filesystem, logs won’t be persisted on the Pi after a reboot or power loss, so remote logging is very helpful for troubleshooting.

microSD card choice

For this use case, using the smallest SD card possible is fine; a high-endurance card is ideal but not strictly necessary, since the whole point is to (almost) never write to it.

Plan overview

The overall idea here is to:

• Remove unneeded software & disable unneeded services
• Be sure the Pi is not using its SD card for swap space
• Implement various workarounds and hacks for certain services that expect to be able to write to disk
• Provide tmpfs filesystems for paths that must be writable and store transient data
• Configure the system to mount the root filesystem as read-only
• Add some systemwide shell shortcuts to ease system maintenance

Disable unneeded software and SD card swap

Some read-only Pi guides recommend removing logrotate and using busybox-syslogd instead; I want to keep using journalctl and friends as I’m used to, so I don’t do that.

1. Be sure your Pi isn’t using the SD card for swap space.
2. Look at the services running on the Pi and disable anything you don’t need.

Run an update and reboot

Not strictly necessary, but I like to make sure the system is up to date before freezing it in place:

sudo apt update && sudo apt upgrade
sudo apt autoremove --purge
sudo reboot now

In /boot/cmdline.txt, disable swap and filesystem checks

1. Edit this file via sudo nano /boot/cmdline.txt
2. Append the following: fsck.mode=skip noswap (unless you plan to use an external drive as swap, in which case, omit noswap here)

The resulting line will look something like this (copied from an Pi Zero W):

console=serial0,115200 console=tty1 root=PARTUUID=76b4450a-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait fsck.mode=skip noswap

Migrate to ntp instead of systemd-timesyncd

According to The Internet, systemd-timesyncd won’t work with a read-only filesystem, but we can get ntp to with a few workarounds. We’ll also allow fake-hwclock to write to the filesystem, which isn’t ideal, but the clock resetting back to 1970 on each boot will cause problems.

This is also a good opportunity to use sudo raspi-config to be sure your timezone is set correctly.

We’ll migrate from systemd-timesyncd to ntp:

sudo systemctl disable systemd-timesyncd.service
sudo apt install ntp

We have a few ntp settings to adjust. First, edit /etc/ntpsec/ntp.conf. (On pre-Bookworm releases, this file won’t exist; edit /etc/ntp.conf instead.) Change the driftfile setting to store this state in /var/tmp (which we’ll put in a tmpfs later). The file will then start something like this:

$ head -n 4 /etc/ntp.conf
# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help

driftfile /var/tmp/ntp.drift

Then, enable ntp, via sudo systemctl enable ntpsec (or, on pre-Bookworm releases, sudo systemctl enable ntp).

Next, we need to edit the ntp systemd unit file to avoid using a systemd feature (PrivateTmp) that won’t work on a read-only filesystem. Run sudo systemctl edit ntp, and paste the following lines:

[Service]
PrivateTmp=false

Those will be the only lines in that file.

(Thanks to Christy O’Reilly for providing NTP updates for Bookworm!)

Edit /etc/cron.hourly/fake-hwclock, a script which saves the current clock periodically in case of power failure. This is the one thing that we’re going to allow to write to the SD card. Add the two mount ... lines you see below, so the resulting file looks like this:

#!/bin/sh
#
# Simple cron script - save the current clock periodically in case of
# a power failure or other crash

if (command -v fake-hwclock >/dev/null 2>&1) ; then
  mount -o remount,rw /
  fake-hwclock save
  mount -o remount,ro /
fi

NetworkManager

Note that we will move resolv.conf to /var/run, which will allow NetworkManager update it when needed, but means it’ll be deleted every time the system shuts down. By default, though, NetworkManager won’t touch /etc/resolv.conf if it’s a symlink. To allow NetworkManager to recreate resolv.conf when the system restarts, we need to update /etc/NetworkManager/NetworkManager.conf and add rc-manager=file under the [main] section.

Here’s a sample of a complete, updated /etc/NetworkManager/NetworkManager.conf from one of my systems:

[main]
plugins=ifupdown,keyfile
rc-manager=file

[ifupdown]
managed=false

We’ll now move some files that need to remain writable to /var/run (which is already a tmpfs) and create symlinks from their original locations:

sudo mv /etc/resolv.conf /var/run/resolv.conf && sudo ln -s /var/run/resolv.conf /etc/resolv.conf
sudo rm -rf /var/lib/dhcp && sudo ln -s /var/run /var/lib/dhcp
sudo rm -rf /var/lib/NetworkManager && sudo ln -s /var/run /var/lib/NetworkManager

I won’t lie: I am nervous every time I run this. As with everything in this guide, YMMV.

Move the random-seed file to a writable location

We’ll move the existing systemd random-seed file to a path we’ll put on a tmpfs, and link to it from the original location:

sudo mv /var/lib/systemd/random-seed /tmp/systemd-random-seed && sudo ln -s /tmp/systemd-random-seed /var/lib/systemd/random-seed

To create this file in the /tmp folder at boot before starting the random-seed service, edit the file service file to add an ExecStartPre command. Run sudo systemctl edit systemd-random-seed.service, and paste these lines in:

[Service]
ExecStartPre=/bin/echo "" >/tmp/systemd-random-seed

Note: snapd

I don’t have much experience yet with how snaps behave on a read-only filesystem. So far, this is the behavior I’ve noticed:

1. Programs installed via snap still seem to run.
2. There are messages in the journal like cannot run daemon: fatal: error opening lock file: open /var/lib/snapd/state.lock: read-only file system, but I think these can be safely ignored: there’s no need to hold that lock if snapd can’t write to state.json.
3. Programs that snap had refreshed recently, but which I hadn’t run in a long time, print a warning when I run them: 2024/03/29 16:44:24.831251 cmd_run.go:1046: WARNING: cannot create user data directory: cannot update the 'current' symlink of "/home/cdzombak/snap/go/current": remove /home/cdzombak/snap/go/current: read-only file system. But they still seem to run as expected.

To solve that last annoyance, you can do this before making the filesystem read-only:

1. Update snaps via sudo snap refresh
2. For each snap listed in ~/snap, run the relevant program. On this particular system, this just meant running go and golangci-lint.

If all the snap names in ~/snap are equivalent to binary names, you could do this via the bash one-liner for n in ~/snap/*; do [ -x /snap/bin/"$(basename $n)" ] && /snap/bin/"$(basename $n)"; done.

I reserve the right to update this advice as I learn more, of course!

Disable systemd-rfkill

I can’t find much straightforward discussion on this service and its relationship to the rfkill tool. But, assuming your wireless devices (WiFi, Bluetooth) are currently working as desired, it seems safe to disable this service.

sudo systemctl disable systemd-rfkill.service
sudo systemctl mask systemd-rfkill.socket

Disable daily apt and mandb tasks

Both of these expect to be able to make filesystem writes that persist across reboots. We don’t need them on a system whose software is frozen in place:

sudo systemctl mask man-db.timer

sudo systemctl mask apt-daily.timer
sudo systemctl mask apt-daily-upgrade.timer

Move temporary folders to tmpfs

Finally, we get to the point that we’re adding tmpfs entries to our fstab.

Edit /etc/fstab to include these lines:

tmpfs  /tmp      tmpfs  defaults,noatime,nosuid,nodev   0  0
tmpfs  /var/tmp  tmpfs  defaults,noatime,nosuid,nodev   0  0

Move some spool folders to tmpfs

Edit /etc/fstab to include these lines:

tmpfs  /var/spool/mail  tmpfs  defaults,noatime,nosuid,nodev,noexec,size=25m  0  0
tmpfs  /var/spool/rsyslog  tmpfs  defaults,noatime,nosuid,nodev,noexec,size=25m  0  0

(Note that if you followed my guide to setting up rsyslog on a Pi, there should already be an entry placing /var/spool/rsyslog in a tmpfs.)

Deal with /var/log

We’ll add another tmpfs to /etc/fstab for the /var/log folder:

tmpfs  /var/log  tmpfs  defaults,noatime,nosuid,nodev,noexec,size=50m  0  0

When storing /var/log in RAM, unless you’ve disabled journald, you need to limit the amount of space journald is allowed to use. To do that, edit /etc/systemd/journald.conf. Uncomment the SystemMaxUse=... line (if necessary), and set it to half of your /var/log tmpfs size, or maybe a little less:

#  This file is part of systemd.
# <output snipped by cdzombak>
# See journald.conf(5) for details.

[Journal]
# <output snipped by cdzombak>
SystemMaxUse=25M
# <output snipped by cdzombak>

Optional: Completely disable journald persistence

Instead of moving /var/log to a tmpfs, you might want to configure your system not to write to logs to disk or RAM, particularly if you’ll send logs to a remote syslog server.

To do that, edit /etc/systemd/journald.conf. Uncomment the Storage=... line (if necessary), and change it to Storage=none:

#  This file is part of systemd.
# <output snipped by cdzombak>
# See journald.conf(5) for details.

[Journal]
Storage=none
# <output snipped by cdzombak>

Move logrotate state to tmpfs

logrotate stores some state in /var/lib/logrotate and may not work if it can’t update that folder. Again, add this line to /etc/fstab:

tmpfs  /var/lib/logrotate  tmpfs  defaults,noatime,nosuid,nodev,noexec,size=1m,mode=0755  0  0

Move sudo state to tmpfs

sudo stores some state in /var/lib/sudo, which should be writable. Add this line to /etc/fstab:

tmpfs  /var/lib/sudo  tmpfs  defaults,noatime,nosuid,nodev,noexec,size=1m,mode=0700  0  0

Add ro to the end of your /boot/cmdline.txt line

(Almost there!)

Edit /boot/cmdline.txt again, and append ` ro` to the line.

Modify fstab options to set filesystems as read-only

Edit /etc/fstab again. This time, change the lines that refer to your SD card. In column 4, after the word defaults (without adding any whitespace):

• Add the ,ro flag to both SD card mounts
• If it’s not there already, add the ,noatime option to the / mount

Sample files at this point

/etc/fstab:

proc            /proc           proc    defaults          0       0

PARTUUID=76b4450a-01  /boot           vfat    defaults,ro          0       2
PARTUUID=76b4450a-02  /               ext4    defaults,noatime,ro  0       1

tmpfs  /tmp      tmpfs  defaults,noatime,nosuid,nodev   0  0
tmpfs  /var/tmp  tmpfs  defaults,noatime,nosuid,nodev   0  0
tmpfs  /var/log  tmpfs  defaults,noatime,nosuid,nodev,noexec  0  0
tmpfs  /var/spool/mail  tmpfs  defaults,noatime,nosuid,nodev,noexec,size=25m  0  0
tmpfs  /var/spool/rsyslog  tmpfs  defaults,noatime,nosuid,nodev,noexec,size=25m  0  0
tmpfs  /var/lib/logrotate  tmpfs  defaults,noatime,nosuid,nodev,noexec,size=1m,mode=0755  0  0
tmpfs  /var/lib/sudo  tmpfs  defaults,noatime,nosuid,nodev,noexec,size=1m,mode=0700  0  0

/boot/cmdline.txt:

console=serial0,115200 console=tty1 root=PARTUUID=76b4450a-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait fsck.mode=skip noswap ro

Add systemwide bash integration

Add the following lines to the end of /etc/bash.bashrc:

set_bash_prompt(){
    fs_mode=$(mount | sed -n -e "s/^\/dev\/.* on \/ .*(\(r[w|o]\).*/\1/p")
    PS1='\[\033[01;32m\]\u@\h${fs_mode:+($fs_mode)}\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
}
PROMPT_COMMAND=set_bash_prompt

alias ro='sudo mount -o remount,ro / ; sudo mount -o remount,ro /boot/firmware'
alias rw='sudo mount -o remount,rw / ; sudo mount -o remount,rw /boot/firmware'

(On pre-Bookworm releases, the last two lines should end in /boot, not /boot/firmware.)

This gives you the following features:

• A prompt indicating whether you’re in read-only or read-write mode
• The commands rw to switch to read-write mode, and ro to switch back to read-only mode

Use bash_logout to switch to read-only mode when you log out

Edit this file via sudo nano /etc/bash.bash_logout. It may not exist yet, in which case saving this file from nano will create it. The file should contain this line:

sudo mount -o remount,ro / ; sudo mount -o remount,ro /boot/firmware

(On pre-Bookworm releases, that line should end in /boot, not /boot/firmware.)

Reboot, Verify with mount, Check journalctl for issues

sudo reboot now
# and then wait; SSH back in when the system comes back up

mount
# verify that SD card partitions are mounted `ro`

sudo journalctl -b 0
# scroll through and look for any issues

When looking for issues, you’ll undoubtedly see some errors from various processes. You’ll want to investigate those.

Start by checking “is this actually broken?”. Often there will be messages from e.g. avahi-daemon or snapd that are unhappy they can’t go about their business normally on a read-only filesystem. But as long as that software is still working for your purposes, you can safely ignore their complaints.

References and Acknowledgements

See the notes on my earlier post.

See Also: Considerations for a long-running Raspberry Pi.