Deploying Step CA as SSH CA and simplifying Homelab Access

Deploying a SSH CA involves 3 parties.

1. A server for deploying the SSH CA
2. A server that users want to access
3. The client machine that needs access to server

This process is usually tedious and involves copious usage of arcane openssl commands. I always wanted to implement this, but the work involved in setting this up as well as the headache involved in maintaining it meant that this was out of reach for my homelab.

Recently, I read about Smallstep, an open-core company, that aims to tackle this issue, among other things. It focuses on X.509 certificates primarily, but also acts as a SSL CA.

I won't go too much in-depth about why SSH CA's are interesting and useful, SmallStep does a wonderful job of that on its home page. I am here to show how to implement that setup for a HomeLab or a small-scale self-hosted setup. This is something I find missing in the User Docs by SmallStep. I hope that I can encourage more usage of SSL certificate as I truly think that its a wonderful idea.

This tutorial, you could say, is arranged into 2 sections

1. Setting up SmallStep CA as a service on a VPS(Using docker)
2. Configuring servers to accept SSH certificates as authentication method
3. Installing client application on user's machine to enable certificate based SSH login.

Setup Small CA on a VPS

I am assuming that we are working with a "blank" Ubuntu VPS here. By that I mean, a simple install of Ubuntu that services like DigitalOcean provides with their Droplets. The instructions will remain largely similar for other distros, varying in linux user setup and docker install.

P.S. Need to open ports TODO

Step 1: Install Docker

Please refer to https://docs.docker.com/engine/install/ubuntu/ for install instructions of Docker

Make sure to also complete the Post Installation steps described here for QoL improvements

Step 2: Create App User

We need to start with creation of app user. These services usually only provide blank installs with a root user. It is inadvisable to deploy applications as root due to security concerns

# Create App user w/ home directory
sudo useradd -m app

# Optional. Don't do this if you need maximum security.
sudo passwd app      

# Add app user to docker for accessing docker with sudo permission
sudo usermod -aG docker app

Step 3: Configure Step CA

We will be putting all configuration of Step CA at ~/apps/step_ca under the home directory of app user. Unless mentioned otherwise, please run the commands as app user.

# Create a empty directory for storing StepCA config
mkdir -p ~/apps/step_ca/config

Go to step_ca dir and execute the initialization command

In this step, step command asks for the following Info

1. Deployment Type -> Standalone (this means self-hosted)
2. CA Name -> Whatever name you want as the authority on your certificates
3. DNS Names/IP Addresses -> This should be the DNS name of the server that runs Step CA. For a localhost installation, this can be localhost
4. Port -> I choose :8443 as I will be running other HTTPS services on this server as well. Due to security concerns, it is impossible to run SmallStep behind L7 proxy(the default mechanism of Nginx, Apache, Caddy) and not suggested to run behind L4 proxies.
5. Provisioner -> your email works here, not necessary though
6. Password -> Make sure that this is a secure password. It is of great import that this password be easy to type and remember if you don't setup an alternative provisioner. You will be typing this password way too many times in that case.

app@server01:~/apps/step_ca$ docker run -it -v $(pwd)/config:/home/step smallstep/step-ca step ca init --ssh
there is no ca.json config file; please run step ca init, or provide config parameters via DOCKER_STEPCA_INIT_ vars
✔ Deployment Type: Standalone
What would you like to name your new PKI?
✔ (e.g. Smallstep): CA_NAME
What DNS names or IP addresses would you like to add to your new CA?
✔ (e.g. ca.smallstep.com[,1.1.1.1,etc.]): ca.example.com
What IP and port will your new CA bind to?
✔ (e.g. :443 or 127.0.0.1:443): :8443
What would you like to name the CA's first provisioner?
✔ (e.g. you@smallstep.com): mail@example.com
Choose a password for your CA keys and first provisioner.
✔ [leave empty and we'll generate one]:
✔ Password: FwKPHWTYn9jL9nFxNg2EfuC8A

Generating root certificate... done!
Generating intermediate certificate... done!
Generating user and host SSH certificate signing keys... done!

✔ Root certificate: /home/step/certs/root_ca.crt
✔ Root private key: /home/step/secrets/root_ca_key
✔ Root fingerprint: 0e2146facbe44cc782ae5abcdef8541757646623456723105b1ac4285754768a
✔ Intermediate certificate: /home/step/certs/intermediate_ca.crt
✔ Intermediate private key: /home/step/secrets/intermediate_ca_key
✔ SSH user public key: /home/step/certs/ssh_user_ca_key.pub
✔ SSH user private key: /home/step/secrets/ssh_user_ca_key
✔ SSH host public key: /home/step/certs/ssh_host_ca_key.pub
✔ SSH host private key: /home/step/secrets/ssh_host_ca_key
✔ Database folder: /home/step/db
✔ Templates folder: /home/step/templates
✔ Default configuration: /home/step/config/defaults.json
✔ Certificate Authority configuration: /home/step/config/ca.json

Your PKI is ready to go. To generate certificates for individual services see 'step help ca'.

FEEDBACK 😍 🍻
  The step utility is not instrumented for usage statistics. It does not phone
  home. But your feedback is extremely valuable. Any information you can provide
  regarding how you’re using `step` helps. Please send us a sentence or two,
  good or bad at feedback@smallstep.com or join GitHub Discussions
  https://github.com/smallstep/certificates/discussions and our Discord
  https://u.step.sm/discord.

Out of this wall of text, the key info here is the Root Fingerprint. This is required for bootstrapping the clients and host servers with the CA certificates. Refer to SmallStep documentation to figure out what is the need for the bootstrap process.

 Root fingerprint: 0e2146facbe44cc782ae5abcdef8541757646623456723105b1ac4285754768a

Move to ~/apps/step-ca and create docker-compose.yml with the following configuration.

version: "3.9"
services:
  ca:
    image: smallstep/step-ca
    ports:
      - "8443:8443"
    volumes:
      - ./config:/home/step
    restart: always

Start the service using the following command. As long as docker post installation steps are carried, there will be no need for sudo here.

docker compose up -d

You can check the logs like so

docker compose logs
 ⠿ Container step_ca-web-1  Started                                                                                                                                           0.6s
app@server01:~/apps/step_ca$ docker compose logs
step_ca-web-1  | badger 2022/09/29 21:20:04 INFO: All 0 tables opened in 0s
step_ca-web-1  | 2022/09/29 21:20:04 Starting Smallstep CA/0.22.1 (linux/amd64)
step_ca-web-1  | 2022/09/29 21:20:04 Documentation: https://u.step.sm/docs/ca
step_ca-web-1  | 2022/09/29 21:20:04 Community Discord: https://u.step.sm/discord
step_ca-web-1  | 2022/09/29 21:20:04 Config file: /home/step/config/ca.json
step_ca-web-1  | 2022/09/29 21:20:04 The primary server URL is https://ca.example.com:8443
step_ca-web-1  | 2022/09/29 21:20:04 Root certificates are available at https://ca.example.com:8443/roots.pem
step_ca-web-1  | 2022/09/29 21:20:04 X.509 Root Fingerprint: 0e2146facbe44cc782ae5abcdef8541757646623456723105b1ac4285754768a
step_ca-web-1  | 2022/09/29 21:20:04 SSH Host CA Key: ecdsa-sha2-nistp256 0e2146facbe44cc930ae5abcdef8541757646623456723105b1ac4285754768a/00SgzlfAZSjCQDutBjrheDycoMFf4AUhVnJdey9xbNolEDHStfRvchgXtK3Gl+hdey9xbNSILm+v9KM=
step_ca-web-1  | 2022/09/29 21:20:04 SSH User CA Key: ecdsa-sha2-nistp256 0e2146facbe44cc930ae5abcdef8dey9xbN23456723105b1ac4285754768aS2u5VDvZ1XU+VaXMAkuBXcxIyX2/Xt/Hdey9xbN+dey9xbNYg2SA=
step_ca-web-1  | 2022/09/29 21:20:04 Serving HTTPS on :8443 ...

Step 4: Open the port 8443 on the Hosting platforms Dashboard.

Usually, the port will not be open by default and any attempt to access it will be blocked by the firewall of the provider.

Configuring servers to accept SSH certificates as authentication

Step 1: Bootstrapping the SSH CA on the server

This process needs root permissions on the server. The root permission is required to add the CA certificate to the list of trusted certificate list of the operating system.

step ca bootstrap --ca-url https://ca.example.com:8443 --fingerprint 0e2146facbe44cc782ae5abcdef8541757646623456723105b1ac4285754768a --install

Step 2: Create host certificates for server

I am creating the certificates in /etc/ssh as that eliminates a copy step. Make sure that you backup essential certificate material if needed before this step. The --host parameter should be domain of the server for which certificate is been granted.

It is essential that the parameters --insecure and --no-password are present here as the sshd daemon needs to start without user input and putting passphrase is just asking for trouble. [You will find the sshd daemon unable to start. Guess how I know that ;-( ]

cd /etc/ssh

# might have to overwrite some keys. Make sure to have backups of the keys here
step ssh certificate --insecure --no-password --host blog.example.com ssh_host_ecdsa_key 

# Trust the SSH CA public cert on the server as well as set up this server's host certificate
step ssh config --host \
  --set Certificate=ssh_host_ecdsa_key-cert.pub \
  --set Key=ssh_host_ecdsa_key

Installing client application on user's machine to enable certificate based SSH login.

Bootstrap the CA just as you would do on server-side. Again, root permissions are required to modify the trust store.

root@personal:~# step ca bootstrap --ca-url https://ca.example.com:8443 --fingerprint 0e2146facbe44cc782ae5abcdef8541757646623456723105b1ac4285754768a--install
The root certificate has been saved in /root/.step/certs/root_ca.crt.
The authority configuration has been saved in /root/.step/config/defaults.json.
Installing the root certificate in the system truststore... done.

Accessing Servers without passwords and SSH key pairs

After all this effort, now is the time to enjoy the fruits of our labor.

$ eval `ssh-agent`
$ step ssh login app
✔ Provisioner: mail@example.com (JWK) [kid: n73UsHnNxl55CGpQp0frad32432Z1bPyn7gq7Eg]
Please enter the password to decrypt the provisioner key: 





























✔ CA: https://ca.edwinclement08.com:8443
✔ SSH Agent: yes
$ ssh-add -l
256 SHA256:QzPf6Vi/7E5PU3UYQtZoztsy+/fnvJVYPrAEIUy56Yw edwin2 (ECDSA-CERT)
$ ssh app@server
app@server:~$ 

Here app is the username for which access is granted. The server should have an app user that you will gain access to. In case you need root access, you need to swap out app with root instead.

If you want access to multiple users, then you must use --principal for adding multiple users.

Note: to have multiple principals, you can add --principal arguments

step ssh login --principal root --principal app mail@example.com

Further Goals

This here is just the base level of SSH certificate based authentication. There are many more things to play with, like SSH certificate auto-renewal, SSO-based provisioning, PAM based provisioning, Token based provisioning. This post is already quite long. Will be covering the remainder in a future post.