Securing my personal SSH infrastructure with Yubikeys

February 08, 2021 • Tagged: security-and-privacy linux series:project-logs

Part of the Project Logs series.

One recently-completed project I mentioned in January’s “Now” post was locking down SSH in my personal computing infrastructure using Yubikeys. In this post, I’ll outline my goals, the strategy I took, and the problems and solutions I ran into along the way.

Goals & Strategy

Historically, I’ve used a pretty basic SSH setup for my personal projects: my user account on every laptop/desktop/server had its own key in ~/.ssh, and I’d try to keep the authorized_keys lists on all my servers more-or-less up-to-date. This presents a number of obvious security problems.

I wanted to ensure that, should an attacker gain access to one of my servers, they can’t use that access to move onto any other computer I control. To do this, I moved to using a few Yubikeys to store my SSH keys; there’s no longer key material stored on any server for a hypothetical attacker to steal. The Yubikeys require a PIN, so this is an example of two-factor authentication: something I physically have, and something I know.

SSH agent forwarding is used to allow me to SSH from one server to another or fetch code from GitHub on a remote server. With yubikey-agent, my preferred agent software, every single SSH operation — yes, even those performed via agent forwarding — requires a physical touch to confirm.

I use a private Git repository to synchronize SSH configuration (including authorized_keys, the list of public keys corresponding to my Yubikeys) between machines, with a modular local configuration system allowing me to quickly enable commonly-used SSH configuration blocks which only apply to a subset of my machines.

Implementation

Yubikeys

I found it easiest by far to use yubikey-agent for this project. It’s pretty straightforward to set this up; the real work was figuring out how to smooth out the various difficulties I encountered later.

(The commands and configuration changes under this heading apply to client machines with attached Yubikeys.)

Install yubikey-agent with Homebrew:

brew tap filippo.io/yubikey-agent https://filippo.io/yubikey-agent
brew install yubikey-agent
brew services start yubikey-agent

Run yubikey-agent -setup to generate a new SSH key on your Yubikey.

Set the SSH_AUTH_SOCK environment variable (do this in your .bashrc or .zshrc). In my dotfiles, I first check that yubikey-agent is installed, then proceed:

command -v yubikey-agent >/dev/null 2>&1 && export SSH_AUTH_SOCK="/usr/local/var/run/yubikey-agent.sock"

Enable SSH agent forwarding for a specific, trusted host (don’t enable it for all hosts; that’s a potential security issue) by adding ForwardAgent yes to that host’s block in your ~/.ssh/config. A complete example might look like:

Host dzombak.com
    User chris
    HostName dzombak.com
    ForwardAgent yes

This is a bit of a spoiler for a topic later in this blog post, but we’ll also add this to our ~/.ssh/config:

Host *
    IdentityAgent /usr/local/var/run/yubikey-agent.sock

This allows apps started from outside your terminal — like the GUI Git client, Fork.app — to find and use yubikey-agent.

A note: Secretive.app

I’d like to use the new macOS app Secretive, which stores SSH keys in the Secure Enclave on newer MacBooks and requires Touch ID to authenticate. Unfortunately, for Reasons™ I’m still using macOS Mojave, and Secretive requires Catalina or Big Sur. I plan to move to Big Sur soon enough, since I want to get an M1 MacBook Pro when the 16” models are released, so I’ll be able to try Secretive soon enough. (Worth noting, this changes the security model somewhat, as the second factor is biometric rather than a PIN, but it’s still two factors.)

Server Configuration

Of course, moving to Yubikeys doesn’t solve much if your servers still allow password logins. On every server, in /etc/ssh/sshd_config, I set the following. Make this change only after you’ve set up a Yubikey and added it to authorized_keys for your user account on the server!

ChallengeResponseAuthentication no
PasswordAuthentication no
PermitRootLogin no

(That last line — PermitRootLogin no — ensures that logins as root via SSH are never allowed, which is a good SSH best practice unrelated to Yubikeys.)

Restart the SSH service, and immediately — before logging out — open a new terminal window and test that you can still login to the server with your Yubikey.

macOS tends to lose changes to sshd_config during OS upgrades, so after installing macOS updates I make sure to check that my SSH server configuration is intact.

Git repo for SSH configuration

I keep my ~/.ssh directory, with a few important exceptions (keys are never committed!), in a private Git repo. This allows me to sync configuration and authorized_keys changes between systems easily. I’ve created a stripped-down version at cdzombak/ssh-example which you can use as a basis for your own setup.

I’ll walk through the highlights here:

README.md covers initial installation & setup.
authorized_keys is where you’ll add the public keys associated with the new, private SSH keys on your Yubikeys.
config is where you’ll add Host blocks for your servers. At the top, it sets some SSH best practices that I’ve accumulated over the years.
fix-permissions.sh ensures that ~/.ssh/authorized_keys and ~/.ssh/rc have the correct permissions. I run this after pulling from the repository; if it results in any changes the files’ permissions in the repo should be corrected.
rc runs after I log into a machine via SSH. In this case, it updates a symlink in ~/.ssh/sock to point to the new SSH agent socket. See the “Long-lived screen sessions” section, below, for an explanation on why this is necessary.
config.templates/ contains SSH configuration blocks which can be included on a given machine, but shouldn’t be included everywhere. The most important of these are yubikey-agent, which when enabled sets IdentityAgent to the yubikey-agent socket as described above; and homedir-ssh-auth-sock, which sets IdentityAgent to the symlink rc creates after an SSH login. On any given machine, I enable one and only one of these two configurations, depending whether it’s a client machine with Yubikey attached or a server which will rely on agent forwarding for any outgoing SSH connections.
config.local/ is included by config and ignored by Git. I can add symlinks from here to config.templates to enable a specific SSH configuration block on the machine.

Those last two — config.local and config.templates — are important, because that’s how I achieve variations in SSH configuration between different machines. The README covers how to enable config templates on a given computer.

Challenges & Solutions

Long-lived `screen` sessions

I use GNU screen (yes, still; I haven’t bothered to learn tmux) as a terminal multiplexer and to provide persistence between SSH sessions. This was a problem for SSH agent forwarding: when I first SSH in and start a screen session, the SSH_AUTH_SOCK environment variable would be set. But when I logged in from somewhere else and reattached to the screen session, the SSH_AUTH_SOCK environment variable wouldn’t get updated, so SSH agent forwarding was broken until I started a new screen session.

This Gist helped me fix this situation. There are a few parts to this solution. (The commands and configuration changes under this heading apply to servers you’ll SSH into.)

First, we have to have a location for our SSH agent socket that doesn’t change between logins. These few lines in ~/.ssh/rc achieve this:

if test "$SSH_AUTH_SOCK" ; then
    ln -sf "$SSH_AUTH_SOCK" ~/.ssh/sock/ssh_auth_sock
fi

Great! Then we just need clients to use this new, always-updated socket. To do this, we configure ~/.screenrc to set the environment variable SSH_AUTH_SOCK:

setenv SSH_AUTH_SOCK $HOME/.ssh/sock/ssh_auth_sock

Finally, we’ll also want to include IdentityAgent ~/.ssh/sock/ssh_auth_sock in our SSH configuration. We can do this by including the homedir-ssh-auth-sock configuration block within my modular SSH configuration setup:

cd ~/.ssh/config.local
ln -s ../config.templates/homedir-ssh-auth-sock .

SSH agent forwarding when running commands under sudo

When running a command with sudo, you’re working in a new environment; your user’s environment variables are not preserved. This will, of course, break SSH agent forwarding.

To solve this, we want to preserve the SSH_AUTH_SOCK environment variable when using sudo. (The commands and configuration changes under this heading apply to servers you’ll SSH into.)

Run visudo (as root) to edit your sudoers file, and add:

Defaults>root    env_keep+=SSH_AUTH_SOCK

This means that when using sudo to run a command as root (not as any other user), your SSH_AUTH_SOCK variable remains intact, and agent forwarding works as expected.

SSH agent forwarding via `SSH_AUTH_SOCK` doesn’t work with GUI macOS apps

This issue in the Fork app’s issue tracker was really helpful here. .bashrc and .zshrc don’t apply to GUI apps (unless you launch them from the terminal), so setting SSH_AUTH_SOCK only in shell configuration files won’t work.

This is why we need IdentityAgent /usr/local/var/run/yubikey-agent.sock in our SSH configuration. To enable this within my modular SSH configuration setup:

cd ~/.ssh/config.local
ln -s ../config.templates/yubikey-agent .

(This change applies to macOS client machines with attached Yubikeys.)

Avoiding repeated mystery Yubikey prompts (using HTTPS for GitHub and Bitbucket repositories)

After I set this up, my Yubikey would periodically blink as if I were trying to SSH into something, but I hadn’t tried to do anything with SSH! That was worrying, until I realized it was just Fork trying to update repository info in the background.

I decided to configure Git on my laptops to use HTTPS instead of SSH when communicating with GitHub and Bitbucket, so Fork can work in the background as it desires.

To do this, we’ll add the following to ~/.gitconfig. (This change applies to any Mac where you’d like Git to use HTTPS instead of SSH.)

[url "https://"]
	insteadOf = git://

[url "https://github.com/"]
	insteadOf = git@github.com:

[url "https://bitbucket.org/"]
	insteadOf = git@bitbucket.org:

[credential]
	helper = osxkeychain

Now, when you try to perform a Git operation for a GitHub or Bitbucket repo for the first time, Git will prompt for credentials:

Username for 'https://bitbucket.org':
Password for 'https://bludzombak@bitbucket.org':

For GitHub, your password is a Personal Access Token with repo scope. For Bitbucket, your password is an App Password with Repositories/Write and Repositories/Read permissions.

Git will cache these credentials on macOS’s Keychain for future use.

Servers that need to communicate autonomously

I do have a use case where one server needs to sync data to another periodically using rsync over SSH, meaning a Yubikey that requires physical, real-time interaction is out of the question.

To achieve this, the receiving server has a restricted user account which only allows access to the necessary data. The sending server, who initiates the sync, has an SSH key which is used only for this task; and which can only log into the restricted user account on the receiving server.

For an extra layer of moderate security (it’s still not a real second factor), the receiving server restricts the IP address from which that SSH key can login, using the from directive in its authorized_keys file. See Restricting SSH logins to particular IP addresses and Configuring Authorized Keys for OpenSSH for more details on that.

iPhone SSH clients (Prompt & Secure ShellFish)

I use Prompt 2 on my iPhone occasionally when a laptop isn’t handy. The app generates its own SSH key, which is stored on the iPhone (not in Panic Sync), and I add the corresponding public key to authorized_keys in my SSH configuration repository.

I figure this is fairly secure, because the key stays on my phone and is secured behind Face ID & the iPhone’s PIN, meaning two factors are required: something I have and one of (biometrics or something I know).

It’s a similar story with Secure ShellFish. I use this to remotely access files on my home NAS. In this case, Secure ShellFish’s SSH key actually allows logging in only to a user account on the NAS with limited permissions; it isn’t added to my “core” SSH configuration.

Disaster Recovery

An additional Yubikey lives in a fireproof safe to aid in recovery in case all other Yubikeys are lost or destroyed.

Windows clients (unsolved)

I’m just ignoring this for now. There’s only one Windows machine I use even somewhat regularly, and I never need to SSH into anything from it. It’d be nice to learn about an SSH agent solution that supports Yubikeys or the Google Titan security key, but I have no motivation to work on this myself.