Remote logging for easier Raspberry Pi debugging
Part of the Raspberry Pi Reliability series.
When something does go wrong, having collected logs somewhere other than your Raspberry Pi’s SD card is very helpful. Depending on how your Pi is set up:
- Logs may only be kept for the current boot or may be stored only in RAM, so after power-cycling the Pi you’ll have no log of what happened.
- Or, if SSH isn’t working and you don’t have easy physical access to the Pi, you’re in the same situation: no logs.
I’ve been using Logz.io’s community/free plan at home for several years now; it’s more than adequate for this use case. The remainder of this post will focus on setting up rsyslog with Logz.io, but in any case your takeaway from this post should be “remote log collection is helpful.”
Sending system logs to Logz.io with rsyslog
You should peruse the Logz.io documentation (or the docs for your preferred logging service) to see all your log shipping options, but here’s the TL;DR on how I set this up on my Linux systems:
TMP_DIR=$(mktemp -d 2>/dev/null || mktemp -d -t 'logz-linux')
cd "$TMP_DIR"
curl -sLO https://github.com/logzio/logzio-shipper/raw/master/dist/logzio-rsyslog.tar.gz
tar xzf logzio-rsyslog.tar.gz
sudo rsyslog/install.sh -t linux -a "$MY_LOGZIO_TOKEN" -l "listener.logz.io"
That’s it!
Configuring rsyslog to minimize SD card wear
The default rsyslog setup is fine for anything that’s not running from a microSD card … so I customize it on the Raspberry Pi.
These changes are not without risk, and you should only use any given intervention if you understand it and what it will do. (See my Pi Reliability post on risk vs. benefits.)
First, edit /etc/fstab and create a tmpfs for /var/spool/rsyslog. Feel free to change the 25M to something like 50M if your Pi has the memory headroom:
tmpfs /var/spool/rsyslog tmpfs defaults,noatime,nosuid,nodev,noexec,size=25M 0 0
Then we need to tell rsyslog how much disk space it’s allowed to use. The rsyslog docs specifically warn that it doesn’t handle running out of disk space gracefully.
In /etc/rsyslog.d/22-logzio-linux.conf you’ll notice a line like $ActionQueueMaxDiskSpace 1g. That won’t do, since we just put it in a 25MB filesystem in RAM!
Change that line to make the limit just under the size of your new /var/spool/rsyslog filesystem:
$ActionQueueMaxDiskSpace 24m
Reboot the Pi when you’re done.
Watching for critical errors with Logz.io
Logz.io’s alerts feature is useful, and if you’re sending your Raspberry Pi logs to it you can easily be alerted to a few critical errors. I use this to get alerted to “out of disk space” and “out of memory” errors.
Out of disk space
Set up a new alert, with the following properties:
- Query:
"No space left on device" - Trigger if:
number of logsin the last5minutes is>0, it’shighseverity - Schedule:
always - Notify: add your email address, Ntfy webhook, etc. and customize notification options as desired
Out of memory
Set up another alert, with the following properties:
- Query:
message:"kernel: Out of memory:" - Trigger if:
number of logsin the last5minutes is>0, it’shighseverity - Schedule:
always - Notify: add your email address, Ntfy webhook, etc. and customize notification options as desired