Chris Dzombak

sharing preview • dzombak.com

Reducing SD Card Wear on a Raspberry Pi or Armbian Device

A guide to reducing SD card wear for a more reliable Raspberry Pi installation.

Reducing SD Card Wear on a Raspberry Pi or Armbian Device

• Tagged: raspberry-pi linux series:pi-reliability

Part of the Raspberry Pi Reliability series.

December 29, 2023: This guide has been supplanted by an updated series of blog posts; see my recent post, Considerations for a long-running Raspberry Pi.

Recently, I got very interested in reducing the wear on the MicroSD cards used in various Raspberry Pi (and Orange Pi) deployments around the house. (Props to PiKVM for exposing me to the idea of a read-only Raspberry Pi filesystem in the first place — check out my blog post about my PiKVM build for more on that project.)

This is an advanced guide. Depending on what you’re doing on your Pi, some of these steps could cause data loss, be difficult or impossible to reverse, or require erasing your SD card and reinstalling the OS.

Only follow any given step if you completely understand it and its implications for your Pi.

Using this Guide

I’ve split this guide into several top-level sections. Some (“Choosing an SD Card,” “Disable unneeded services & remove unneeded software”) apply to all systems; others only apply in specific situations. You should choose those sections which apply to your specific application:

Each of these sections will include a list of references, which you can use to learn more about the advice in the section.

The specific steps here are generally written for Raspberry Pi OS 10 (Buster).

The “If the system can’t use a read-only filesystem” and “If the system needs swap” sections also include some notes on Armbian 11 (Bullseye).

Note: Advice to avoid

This Stack Exchange post suggests disabling journaling on the filesystem. While this can reduce wear, it will also make your Pi more likely to face filesystem corruption in the event of a crash or power outage.

My entire goal when researching this guide for myself was to make my Pis more reliable, and disabling journaling works against that goal.

Choosing an SD Card

First, choose a good-quality, fast, SD card from a reputable brand. In recent projects, I’m trying to use SanDisk Industrial MicroSD cards if 16GB is enough for the application; or SanDisk High Endurance Micro SD cards if I need more space.

Use f3 to test the card before putting it into service. On macOS, it can be installed via brew install f3; and then using it looks like this1:

f3write /Volumes/sdcard/ && f3read /Volumes/sdcard/

Disable unneeded services & remove unneeded software

Figure out what’s running via:

sudo systemctl --type=service --state=running

Disable running services you know you don’t need, via sudo systemctl disable name-of-service.service.

In particular, on various Pis I have disabled:

Uninstall some other items that all or most read-only Raspberry Pi guides recommend removing (unless you’re using any of this software):

sudo apt remove --purge wolfram-engine triggerhappy xserver-common lightdm && sudo apt autoremove --purge

Remote Logging

I’ve been sending some system logs to logz.io using their free “community” plan for a while. I signed up while looking for a way to consolidate logs while debugging my Aircraft Radar Alexa Skill, and I’ve found it useful to send logs there from other systems too.

As part of this guide, you’ll end up putting logs in memory at least temporarily; on a read-only filesystem, logs will of course be lost after a crash or shutdown. Sending logs to a remote logging server is one of the only ways to debug what happened in these scenarios.

Once you’re logged into a logz.io account, they provide simple instructions for sending Linux logs to them using rsyslog. That looks something like this, though my API key is censored in this example:

curl -sLO https://github.com/logzio/logzio-shipper/raw/master/dist/logzio-rsyslog.tar.gz \
  && tar xzf logzio-rsyslog.tar.gz \
  && sudo rsyslog/install.sh -t linux -a "v...M" -l "listener.logz.io"

Update: If you're using rsyslog and placed /var/spool/rsyslog in a tmpfs, you'll also want to change $ActionQueueMaxDiskSpace to the size of that tmpfs, minus 1 MB.

For logz.io, this can be changed by editing /etc/rsyslog.d/22-logzio-linux.conf.

If the system can use a read-only filesystem

Example Application: environmental data loggers, which log periodic readings to a remote time-series database and store no state locally.

For this use case, using the smallest SD card possible is fine; a high-endurance or industrial is ideal but not strictly necessary, since the whole point is to (almost) never write to it.

This is a somewhat risky procedure; following these steps, even correctly, could render your Pi unbootable, requiring you to connect a keyboard and monitor to fix it.

If your Pi is installed in a hard-to-reach spot, be prepared to retrieve it if this process goes awry.

The overall idea here is to:

Remove unneeded software

Some read-only Pi guides recommend removing logrotate and using busybox-syslogd instead; I want to keep using journalctl and friends as I’m used to, so I don’t do that.

Be sure you’ve listed running services and disabled anything you don’t need, as described above.

Additionally, dphys-swapfile, which manages a swapfile in the root filesystem on the SD card, won’t be able to work. If the system needs swap, set up a (sacrificial) external USB drive for a swap partition, following the “If the system needs swap” section.

sudo dphys-swapfile swapoff
sudo dphys-swapfile uninstall
sudo update-rc.d dphys-swapfile remove
sudo apt purge dphys-swapfile

Verify with free -m, which should show all 0s in the swap row:

$ free -m
              total        used        free      shared  buff/cache   available
Mem:            430          33         333           6          63         342
Swap:             0           0           0

Run an update and reboot

This is not strictly necessary, but I like to make sure the system is up to date before freezing it in place:

sudo apt update && sudo apt upgrade
sudo apt autoremove --purge
sudo reboot now

In /boot/cmdline.txt, disable swap and filesystem checks

  1. Edit this file via sudo nano /boot/cmdline.txt
  2. Append the following: fsck.mode=skip noswap (unless you plan to use an external drive as swap, in which case, omit noswap here)

The resulting line will look something like this (copied from an Pi Zero W):

console=serial0,115200 console=tty1 root=PARTUUID=76b4450a-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait fsck.mode=skip noswap

Older guides recommend you add fastboot to this line. This has been replaced by fsck.mode=skip.

Migrate to ntp instead of systemd-timesyncd

According to The Internet, systemd-timesyncd won’t work with a read-only filesystem, but we can get ntp to with a few workarounds. We’ll also allow fake-hwclock to write to the filesystem, which isn’t ideal, but the clock resetting back to 1970 on each boot could lead to problems.

This is also a good opportunity to use sudo raspi-config to be sure your timezone is set correctly.

We’ll migrate from systemd-timesyncd to ntp:

sudo systemctl disable systemd-timesyncd.service
sudo apt install ntp

The, we have a few ntp settings to adjust. First, edit /etc/ntp.conf. Change the driftfile setting to store this state in /var/tmp (which we’ll later put in a tmpfs). The file will then start something like this:

$ head -n 4 /etc/ntp.conf
# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help

driftfile /tmp/ntp.drift

We’ll then enable ntp, via sudo systemctl enable ntp.

Next, we need to edit the ntp systemd unit file to avoid using a systemd feature (PrivateTmp) that won’t work on a read-only filesystem. Run sudo systemctl edit ntp, and paste the following lines:

[Service]
PrivateTmp=false

Those will be the only lines in that file.

Edit /etc/cron.hourly/fake-hwclock, a script which saves the current clock periodically in case of power failure. This is the one thing that we’re going to allow to write to the SD card. Add the two mount ... lines you see below, so the resulting file looks like this:

#!/bin/sh
#
# Simple cron script - save the current clock periodically in case of
# a power failure or other crash

if (command -v fake-hwclock >/dev/null 2>&1) ; then
  mount -o remount,rw /
  fake-hwclock save
  mount -o remount,ro /
fi

DHCP & DHCPCD5

We’ll shuffle some networking files around, remove some that aren’t strictly needed, and create symlinks from their original locations to /var/run, which is already a tmpfs:

sudo mv /etc/resolv.conf /var/run/resolv.conf && sudo ln -s /var/run/resolv.conf /etc/resolv.conf
sudo rm -rf /var/lib/dhcp && sudo ln -s /var/run /var/lib/dhcp
sudo rm -rf /var/lib/dhcpcd5 && sudo ln -s /var/run /var/lib/dhcpcd5

I won’t lie: I was nervous when I first ran that on one of my Pis, but everything seemed fine afterward. YMMV.

Move the random-seed file to a writable location

We’ll move the existing systemd random-seed file to a path we’ll put on a tmpfs, and link to it from the original location:

sudo mv /var/lib/systemd/random-seed /tmp/systemd-random-seed && sudo ln -s /tmp/systemd-random-seed /var/lib/systemd/random-seed

To create this file in the /tmp folder at boot before starting the random-seed service, edit the file service file to add an ExecStartPre command. Run sudo systemctl edit systemd-random-seed.service, and paste these lines in:

[Service]
ExecStartPre=/bin/echo "" > /tmp/systemd-random-seed

Disable systemd-rfkill

I can’t find much straightforward discussion on this service and its relationship to the rfkill tool. But, assuming your wireless devices (WiFi, Bluetooth) are currently working as desired, it seems safe to disable this service.

sudo systemctl disable systemd-rfkill.service
sudo systemctl mask systemd-rfkill.socket

This may not apply if you’ve used the rfkill tool on your Pi to explicitly disable/enable a wireless device before. But, in that case, you know more about rfkill than I do, so you should be able to figure out what’s best for your use case.

Disable daily apt and mandb tasks

Both of these expect to be able to make filesystem writes that persist across reboots. We don’t need them on a system whose software is frozen in place:

sudo systemctl mask man-db.timer

sudo systemctl mask apt-daily.timer
sudo systemctl mask apt-daily-upgrade.timer

Move temporary folders to tmpfs

Finally, we get to the point that we’re adding tmpfs entries to our fstab.

Edit /etc/fstab to include these lines:

tmpfs  /tmp      tmpfs  defaults,noatime,nosuid,nodev   0  0
tmpfs  /var/tmp  tmpfs  defaults,noatime,nosuid,nodev   0  0

Move some transient spool folders to tmpfs

Edit /etc/fstab to include these lines:

tmpfs  /var/spool/mail  tmpfs  defaults,noatime,nosuid,nodev,noexec,size=20M  0  0
tmpfs  /var/spool/rsyslog  tmpfs  defaults,noatime,nosuid,nodev,noexec,size=20M  0  0

Deal with /var/log

We’ll add another tmpfs to /etc/fstab for the /var/log folder:

tmpfs  /var/log  tmpfs  defaults,noatime,nosuid,nodev,noexec,size=50M  0  0

Updated December 29, 2023 to add: When storing /var/log in RAM, I’ve found it helpful and sometimes necessary to limit the amount of space journald is allowed to use. To do that, edit /etc/systemd/journald.conf. Uncomment the SystemMaxUse=... line (if necessary), and set it to half of your /var/log tmpfs size, or maybe a little less:

#  This file is part of systemd.
# <output snipped by cdzombak>
# See journald.conf(5) for details.

[Journal]
# <output snipped by cdzombak>
SystemMaxUse=20M
# <output snipped by cdzombak>

Completely disable journald persistence

Instead of moving /var/log to a tmpfs, you might want to configure your system not to write to logs to disk or RAM, particularly if you’ll send logs to a remote syslog server.

To do that, edit /etc/systemd/journald.conf. Uncomment the Storage=... line (if necessary), and change it to Storage=none:

#  This file is part of systemd.
# <output snipped by cdzombak>
# See journald.conf(5) for details.

[Journal]
Storage=none
# <output snipped by cdzombak>

Move logrotate state to tmpfs

logrotate stores some state in /var/lib/logrotate and may not work if it can’t update that folder. Again, add this line to /etc/fstab:

tmpfs  /var/lib/logrotate  tmpfs  defaults,noatime,nosuid,nodev,noexec,size=1m,mode=0755  0  0

Move sudo state to tmpfs

sudo stores some state in /var/lib/sudo, which should be writable. Add this line to /etc/fstab:

tmpfs  /var/lib/sudo  tmpfs  defaults,noatime,nosuid,nodev,noexec,size=1m,mode=0700  0  0

Add ro to the end of your /boot/cmdline.txt line

(Almost there!)

Edit /boot/cmdline.txt again, and append ro to the line.

Modify fstab options to set filesystems as read-only

Edit /etc/fstab again. This time, change the lines that refer to your SD card. In column 4, after the word defaults (but without adding any whitespace):

Sample files at this point

fstab:

proc            /proc           proc    defaults          0       0

PARTUUID=76b4450a-01  /boot           vfat    defaults,ro          0       2
PARTUUID=76b4450a-02  /               ext4    defaults,noatime,ro  0       1

tmpfs  /tmp      tmpfs  defaults,noatime,nosuid,nodev   0  0
tmpfs  /var/tmp  tmpfs  defaults,noatime,nosuid,nodev   0  0
tmpfs  /var/log  tmpfs  defaults,noatime,nosuid,nodev,noexec  0  0
tmpfs  /var/spool/mail  tmpfs  defaults,noatime,nosuid,nodev,noexec,size=20M  0  0
tmpfs  /var/spool/rsyslog  tmpfs  defaults,noatime,nosuid,nodev,noexec,size=20M  0  0
tmpfs  /var/lib/logrotate  tmpfs  defaults,noatime,nosuid,nodev,noexec,size=1m,mode=0755  0  0
tmpfs  /var/lib/sudo  tmpfs  defaults,noatime,nosuid,nodev,noexec,size=1m,mode=0700  0  0

/boot/cmdline.txt:

console=serial0,115200 console=tty1 root=PARTUUID=76b4450a-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait fsck.mode=skip noswap ro

Add systemwide bash integration

Add the following lines to the end of /etc/bash.bashrc:

set_bash_prompt(){
    fs_mode=$(mount | sed -n -e "s/^\/dev\/.* on \/ .*(\(r[w|o]\).*/\1/p")
    PS1='\[\033[01;32m\]\u@\h${fs_mode:+($fs_mode)}\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
}
PROMPT_COMMAND=set_bash_prompt

alias ro='sudo mount -o remount,ro / ; sudo mount -o remount,ro /boot'
alias rw='sudo mount -o remount,rw / ; sudo mount -o remount,rw /boot'

This gives you the following features:

Use bash_logout to switch to read-only mode when you log out

Edit this file via sudo nano /etc/bash.bash_logout. It may not exist yet, in which case saving this file from nano will create it. The file should contain this line:

sudo mount -o remount,ro / ; sudo mount -o remount,ro /boot

Reboot, Verify with mount, Check journalctl for issues 

sudo reboot now
# and then wait; SSH back in when the system comes back up

mount
# verify that SD card partitions are mounted `ro`

sudo journalctl -b 0
# scroll and look for any issues

Ignore this Avahi issue

When checking logs on my Pi Zero W, I noticed this error:

avahi-daemon[258]: avahi-daemon 0.7 starting up.
avahi-daemon[258]: Successfully called chroot().
avahi-daemon[258]: Successfully dropped remaining capabilities.
avahi-daemon[264]: chroot.c: open() failed: No such file or directory
avahi-daemon[258]: Failed to open /etc/resolv.conf: Invalid argument

This thread describes a similar issue. I’m not sure, in this case, whether the cause is the same (a startup order issue), but the daemon seems to work fine anyway, so I’m ignoring this for now.

Alternatively, you can disable avahi-daemon with sudo systemctl disable avahi-daemon, but then you won’t be able to access your Pi at my-pi-hostname.local.

References (read-only filesystem)

If the system can’t use a read-only filesystem

Example Application: Pi-Hole, which stores settings and state on the filesystem.

For this use case, choose an industrial-grade SD card if possible; otherwise (if more disk space is required) use a high-endurance card. Choose a card with more space than needed, allowing writes to be spread out across the card over time. Apply the optimizations discussed below, with the overall aim to reduce the amount written to the card.

Migrate as much as possible to tmpfs

We want the following to be stored in RAM using tmpfs, not on the SD card:

Run mount first, along with cat /etc/fstab, to check which folders are already in a tmpfs.

Armbian, for example, puts /tmp in RAM by default, while Raspberry Pi OS des not. On both Armbian and Raspberry Pi OS, /run is stored in RAM already; and /var/run links to there.

Edit /etc/fstab and add the following lines (though on Armbian, as noted above, you can skip adding the line for /tmp):

tmpfs  /tmp  tmpfs  defaults,noatime,nosuid,nodev  0  0
tmpfs  /var/spool/rsyslog  tmpfs  defaults,noatime,nosuid,nodev,noexec,size=50M  0  0

This Stack Exchange thread notes that /var/tmp really shouldn't be discarded on reboots, and I don't see much there on any of my Pis, so I've opted to leave it on the SD card.

You could opt to use a tmpfs for it, too, if you'd like.

Find the biggest offenders in /var/log 

sudo du -hs /var/log/* | sort -rh | head -n 5

From here, reduce or disable logging for these services. This is particularly important if you opt to write logs to disk; less so if you log only to RAM.

If you plan to use log2ram, which will keep logs in RAM and occasionally sync them to disk, you should get the size of /var/log under 40MB. This size limit can be increased, but that’s not ideal, especially on something like a Pi Zero with limited RAM.

One easy way to remove (some) historical logs:

sudo rm /var/log/*.1 /var/log/*.gz /var/log/apt/*.gz

Deal with /var/log

On Raspberry Pi OS, /var/log is often one of the biggest culprits for SD card wear. To deal with this, you can either:

  1. Move it entirely to RAM with a tmpfs, as described in the read-only filesystem section above
  2. Use log2ram to log to RAM and occasionally flush to disk, as described below.

Armbian already uses something like `log2ram` by default; logs are written to RAM and only flushed to the SD card occasionally.

You can skip the rest of this section if your device is running Armbian.

Install log2ram

In the following echo ... | sudo tee ... command, replace bullseye by the name of the Debian version your Pi is running. Use lsb_release -a to find this; for example, this is from one of my Pis:

$ lsb_release -a | grep -i codename
No LSB modules are available.
Codename:	stretch

So, on that particular Pi, I would use "deb http://packages.azlux.fr/debian/ stretch main instead.

To install log2ram:

echo "deb http://packages.azlux.fr/debian/ bullseye main" | sudo tee /etc/apt/sources.list.d/azlux.list
wget -qO - https://azlux.fr/repo.gpg.key | sudo apt-key add -
sudo apt update
sudo apt install log2ram

Reboot the Pi. Then verify it’s working, per the documentation. These commands will tell you whether log2ram is running, and allow you to read its logs:

sudo systemctl status log2ram
sudo journalctl -u log2ram -e

And then these commands will allow you to see that the log2ram mount is setup correctly. Sample output from one of my Pis is included:

$ df -h | grep log2ram
log2ram          40M  532K   40M   2% /var/log

$ mount | grep log2ram
log2ram on /var/log type tmpfs (rw,nosuid,nodev,noexec,relatime,size=40960k,mode=755)

Modify fstab options for your root partition (/)

Edit /etc/fstab again. This time, change the lines that refer to your SD card. In column 4, after the word defaults (but without adding any whitespace):

Enforce a filesystem check on every boot

On my Pis, unless a fast startup process is absolutely critical, I like to run fsck on every boot, allowing SD card problems to be caught early (and allowing repairable issues to be repaired).

Run mount | grep "on / " to get the name of the device containing your root filesystem. In this sample output, it’s mmcblk0p1:

$ mount | grep "on / "
/dev/mmcblk0p1 on / type ext4 (rw,noatime,errors=remount-ro,commit=600)

Then, run sudo tune2fs -c 1 /dev/mmcblk0p1, replacing mmcblk0p1 with the name you found above.

To verify that this worked, run cat /etc/fstab, and verify that the root filesystem has a positive integer in the sixth (last) column.

References (read-write filesystem)

If the system needs swap

If this system needs swap to function correctly, we’re going to use an external USB drive for it.

This will absolutely eventually kill the USB drive, but at least that’ll be easy to replace without rebuilding the entire Pi.

Run cat /boot/cmdline.txt. If the line includes noswap, edit /boot/cmdline.txt and remove noswap.

Connect the USB drive and use lsblk to figure out its device name. On the Pi I’m using, this is /dev/sda; if yours differs, use its name in place of /dev/sda in the following commands.

Run sudo cfdisk /dev/sda and:

  1. Select gpt if prompted to select a partition table type.
  2. Create a new primary Linux filesystem partition.
  3. Write & quit

(This article covers those steps in more detail, including screenshots. If you follow that guide, stop after step 7; don’t proceed to mkfs.ext4.)

We’re going to turn that partition into a classic Linux swap partition using mkswap. Run the following command, and note the UUID included in its output:

# sudo mkswap -f /dev/sda1
Setting up swapspace version 1, size = 7.3 GiB (7810555904 bytes)
no label, UUID=1c1b6b47-ad57-43ad-a6ab-f1c2a64a88d1

Then, edit /etc/fstab to add a line for this new swap partition. Replace the UUID below with the one from the previous step:

UUID=1c1b6b47-ad57-43ad-a6ab-f1c2a64a88d1 swap	swap	defaults	0	0

You can enable the swap partition immediately with sudo swapon /dev/sda1, or reboot the Pi and wait. In either case, verify it’s operational using the free command:

$ free -mh
               total        used        free      shared  buff/cache   available
Mem:           999Mi       231Mi       522Mi        42Mi       244Mi       700Mi
Swap:          7.8Gi          0B       7.8Gi

On Raspberry Pi OS, we’ll go ahead and disable the service that manages the SD card-based swap file. We no longer need it if we’re using an external drive, and it will cause additional SD card wear, which we want to avoid:

sudo dphys-swapfile swapoff
sudo dphys-swapfile uninstall
sudo update-rc.d dphys-swapfile remove

If you’re certain you won’t want to reenable swap in the near future, you can completely remove the dphys-swapfile manager with sudo apt purge dphys-swapfile,

References (swap)

If the system does not need swap

Disable swap on Raspberry Pi OS 

sudo dphys-swapfile swapoff
sudo dphys-swapfile uninstall
sudo update-rc.d dphys-swapfile remove

If you’re certain you won’t want to reenable swap in the near future, you can completely remove the dphys-swapfile manager with sudo apt purge dphys-swapfile.

To verify this change, reboot the Pi and run free -m, which should show all 0s in the swap row:

$ free -m
              total        used        free      shared  buff/cache   available
Mem:            430          33         333           6          63         342
Swap:             0           0           0

Disable swap on Armbian

Armbian does not use any SD card-based swap by default, so unless you’ve customized your installation there’s nothing to disable.

There is a zram-based swap partition, which I recall seeing in /etc/fstab. That is a RAM-based compressed block storage device; using it as swap space effectively tiers your RAM into two parts: fast+uncompressed and slower+compressed. I believe the intent here is to increase the effective available memory on RAM-limited boards without touching the SD card, and on my Orange Pis running Armbian I leave this setup as-is.

References (disabling swap)

  1. An extensive f3 overview is outside the scope of this guide, but it’s a simple program, and its README covers usage well. ↩

  2. I can’t help you with the files specific to your Raspberry Pi application, but you should be able to follow the fstab patterns in this guide to set up tmpfs mounts for directories specific to your use case. ↩