Clan + Yubikeys

All I wanted was to test Data Mesher… 🙃

I had done some local dev testing and created a NixOS module with some VM tests. Pinpox had helped write the Clan module and added some more Clan-specific VM tests. We were ready to start testing this in a real Clan.

Until this point, I wasn’t running Clan. I had always meant to convert, but just never got around to it. So this was as good a reason as any. I fired up the docs, opened up my config and got to work.

But it wasn’t long before I hit a few problems.

Multiple User Keys

I’m a long-time user of SOPS via sops-nix for managing secrets within NixOS configurations, and I like to use it in combination with Yubikeys via AGE plugins. Clan can also use SOPS for secrets management, but it deviates in one important way.

Unlike with SOPS where you configure keys in a .sops.yaml file, Clan takes a more active role in how SOPS keys are configured, introducing concepts like users, groups, and machines. And at the time, the user model didn’t allow for more than one encryption key.

Why is this important?

Well, to protect against loss or failure of a Yubikey, I use more than one. And, just in case they’re all lost or broken, I also use a vanilla age key backed up on paper or in a password manager. Ordinarily, this isn’t a problem with SOPS. But with Clan, it was a non-starter.

So I added support for it. ✅

Age Plugins

Having taken care of multiple user keys, I next found myself dealing with some issues related to AGE plugins.

The first was easy enough to fix, ensuring the Clan CLI was loading any necessary AGE plugins when decrypting with SOPS. I added some packages to an allowlist, tweaked the shell being used when invoking SOPS and added an option allowing a user to specify the plugins they require in their Clan:

{
  inputs.clan-core.url = "https://git.clan.lol/clan/clan-core/archive/main.tar.gz";
  inputs.nixpkgs.follows = "clan-core/nixpkgs";

  outputs =
    { self, clan-core, ... }:
    let
      clan = clan-core.clanLib.buildClan {
        inherit self;

        meta.name = "myclan";

        # Add Yubikey and FIDO2 HMAC plugins
        # Note: the plugins listed here must be available in nixpkgs.
        secrets.age.plugins = [
          "age-plugin-yubikey"
          "age-plugin-fido2-hmac"
        ];

        machines = {
          # elided for brevity
        };
      };
    in
    {
      inherit (clan) nixosConfigurations clanInternals;

      # elided for brevity
    };
}

Recovering Recipients

The second issue required a bit more thought, as it turns out there is no uniform mechanism for recovering the recipient (public key) from a secret key generated by an AGE plugin.

It’s important we can do this because the Clan CLI implements a safety check when creating or updating a secret. If the recipient associated with the currently configured AGE private key is not in the list of recipients when encrypting a secret, we add that recipient to the list.

This helps ensure you do not encrypt a secret which you cannot decrypt.

With vanilla AGE keys this isn’t a problem, since you can recover the recipient with a simple age-keygen -y. But for AGE plugins, we ended up with a workaround in which a user needs to prepend a comment with the recipient key.

For example, this is what your ~/.config/sops/age/keys.txt might look like:

# public key: age1zdy49ek6z60q9r34vf5mmzkx6u43pr9haqdh5lqdg7fh5tpwlfwqea356l
AGE-PLUGIN-FIDO2-HMAC-1QQPQZRFR7ZZ2WCV...

For each private key in this file, we look at the preceding line for a comment containing a recipient of the form age1xxxx. The rest of the line doesn’t really matter. You can prefix it with public key:, recipient: or whatever else.

AGE plugin support: ✅

Success?

By this point, I began porting my configuration over to a new repo I had set up for Clan. I added an initial user key based on age-plugin-fido2-hmac and began fleshing out the configuration for my main desktop machine.

After a while, I remembered that I should add a couple more keys to my user, including my backup key. No big deal, just a simple clan secrets users add-key brian age1xxxxxxx right?

Vars + age-plugin-fido2-hmac = Pain

When adding a new user key, under the hood the Clan CLI is making a call to sops updatekeys. As part of the updatekeys process, sops needs to decrypt the underlying value to then encrypt it for the key we are adding. Nothing unusual so far.

The sting in the tail comes when you understand how Vars organises secrets on the filesystem: one file per secret.

This is in stark contrast to how I had been using sops previously, where I typically grouped a number of secrets into the same file on a per-host or a per-user basis. But where’s the problem you might be asking?

age-plugin-fido2-hmac has no form of PIN caching.

For each decryption I have to enter my PIN and touch my Yubikey. And as you can imagine, any non-trivial setup is going to have more than a handful of secrets. When you also consider this needs to be done for shared secrets as well as for each machine’s secrets 😢…

If I had stayed with age-plugin-fido2-hmac, adding a new user key would have been prohibitively expensive and a short path to some form of repetitive strain injury. I could have used a key which didn’t require PIN entry, but I wasn’t comfortable with that.

In the end, I moved to age-plugin-yubikey, as it supports PIN caching and can also be configured to cache presence checks (up to 15 seconds). There is still an issue with pcscd aggressively powering off my yubikey after 5 seconds and ruining the PIN caching, but for day-to-day usage with Clan it works well enough for now.

PIN caching: ✅

Terminal Multiplexing

By this point, having side-quested long enough, I was feeling like I was on the home straight. But there was one last curveball, which at the time of writing, we still don’t have a good solution for: PIN entry when deploying to multiple machines at once.

When you run clan machines update, the Clan CLI will perform the update process concurrently for each machine in your configuration and blend their terminal outputs into one. This becomes a problem when each one of them may ask you to enter a PIN to unlock your Yubikey.

Whilst annoying and suboptimal, this isn’t necessarily a show-stopper for small Clans; it just means you have to update machines individually.

Longer-term, we’re still kicking around a few ideas. For example, it would be nice if age plugins could be configured to ask for pin entry using something like ssh-askpass.

Concurrent PIN entry: ❌

Summary

It took me longer than I would have hoped to just run Data Mesher in a Clan of my own 😂, but along the way there have been some much-needed improvements to Clan itself and, for me personally, I’m left with a much better understanding of age, Yubikeys and sops.

We will continue to look at ways of improving Yubikey integration with Clan.

And in the meantime, please bear in mind there are still a few rough edges 🙏.