I once locked myself out of a client’s server at 11pm because I reloaded sshd with a typo in the config and closed my only terminal. The fix involved a rescue console, a cold coffee and a long apology. That night taught me more about linux server hardening than any checklist had, and it is why the first rule in this post is about what to do before you change anything.
Short version for the impatient: on a fresh small VPS I do five things. Key-only SSH, no root login, a deny-by-default firewall, fail2ban for the noise, and automatic security updates. That takes about ten minutes, and it removes nearly everything that actually gets small servers owned. If you want the reasoning and the config, keep reading.
I run client apps on cheap VPS boxes, mostly Ubuntu 24.04, and I’m not a security specialist. This is a baseline for a server that hosts a web app, a database and a few containers. It is not a compliance program. If you need CIS-level hardening, the CIS Ubuntu benchmark is the document to start with, and it is much longer than what follows.
Open a second terminal before you touch sshd
Here is the habit that would have saved my night. Log in twice. Keep the second session idle and never close it until you have proven that a brand new connection works. A running SSH session survives an sshd reload, so the idle one is your way back in.
Then validate before you reload. sshd -t parses the config and prints errors without touching the running daemon. It costs one second. I skipped it exactly once.
sudo sshd -t && sudo systemctl reload ssh
If that command prints nothing and returns, the config is syntactically fine. It does not tell you the config is what you meant, which is why the second terminal stays open and you test a fresh login from a third window.
SSH: keys only, no root, short leash
Most of the hardening value lives here, because SSH is the one door every VPS has open to the whole internet from minute one. Watch /var/log/auth.log on a new box for ten minutes and you will see strangers trying admin, ubuntu and root with passwords from a list.
I put my changes in a drop-in file instead of editing the main config. On Ubuntu, /etc/ssh/sshd_config includes everything in /etc/ssh/sshd_config.d/*.conf, and sshd uses the first value it finds for each setting. So the filename matters. Cloud images often ship a file like 50-cloud-init.conf that sets PasswordAuthentication yes, and because the first value wins, a file that sorts after it loses. I name mine 01-hardening.conf so it is read first, and then I check the effective result instead of trusting my eyes.
# /etc/ssh/sshd_config.d/01-hardening.conf
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitRootLogin no
MaxAuthTries 3
LoginGraceTime 20
AllowUsers deploy
My first attempt used a 99 prefix and silently lost to the cloud-init file. So after every change I run sudo sshd -T | grep -Ei 'passwordauth|permitroot|maxauth', which prints the settings sshd will actually use. I got this wrong for a while by reading the files instead of asking the daemon, and the daemon disagreed with me twice.
A few notes on those lines, all checked against the sshd_config man page. PermitRootLogin already defaults to prohibit-password, so root can’t log in with a password out of the box, but setting it to no removes the question entirely. MaxAuthTries defaults to 6, and I drop it to 3 because I authenticate with a key and never need six attempts. KbdInteractiveAuthentication no matters because with PAM enabled, keyboard-interactive can sneak password logins back in even when PasswordAuthentication is off.
AllowUsers deploy is the line people skip and I think it deserves more credit. Every account that isn’t named there is refused no matter what else is true. Add a new service account later and forget about it, and it still can’t log in over SSH.
One caveat on moving SSH to another port. I used to do it. It cuts log noise a lot and stops nothing, since a scanner finds port 2222 in seconds. If your logs are loud, fail2ban handles that better and doesn’t break every tool that assumes port 22.
A firewall that denies by default
SSH is not the only thing listening. A fresh image may have a database, a forgotten Redis, a metrics agent or a dev server bound to all interfaces, and you may not remember starting it. A host firewall makes the answer to “what is exposed” a short list you wrote yourself.
On Ubuntu that’s ufw, which ships with the system. The order of commands matters, because enabling a deny-by-default firewall before allowing SSH is another way to lock yourself out.
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw limit OpenSSH
sudo ufw allow 80,443/tcp
sudo ufw enable
sudo ufw status verbose
ufw limit is allow plus a rate limit: it throttles an address that opens too many connections in a short window. It’s not a replacement for fail2ban, but it’s free and one word.
Now the part that bit me. If you run Docker, ufw does not protect your published container ports the way you would assume. Docker writes its own rules into the firewall, and traffic to a published port is diverted before it ever reaches the rules ufw manages. So -p 5432:5432 on a database container can be open to the internet while ufw status swears everything is closed. The Docker documentation on ufw says the two are incompatible in this way and describes the options.
My rule is dull and effective. Publish ports on loopback only (-p 127.0.0.1:5432:5432) unless something truly needs the public interface, and put a reverse proxy in front for the rest. For client sites I went one step further and stopped opening inbound ports at all, which I wrote up in my Cloudflare Tunnel post. If a port isn’t open, there’s nothing to firewall.
After enabling, verify from outside. Run nmap against your own IP from another machine, or use an online port scanner. Looking at the firewall from the inside only tells you what you configured.
Fail2ban for the noise, not for safety
Fail2ban reads logs, spots repeated failed logins and bans the source address in the firewall for a while. I install it on every box, and I try to be honest about what it does. The project’s own README says it reduces the rate of incorrect authentication attempts but cannot eliminate the risk of weak authentication. With key-only SSH, brute force can’t succeed anyway. What fail2ban buys me is quieter logs and less wasted CPU, which also makes real anomalies easier to spot.
The defaults in the upstream jail.conf are a ban time of 10 minutes, a find window of 10 minutes and 5 retries. The sshd jail on Ubuntu is enabled by the packaged defaults, but I still write a small local override instead of editing jail.conf, because package upgrades can overwrite the latter.
# /etc/fail2ban/jail.d/local.conf
[DEFAULT]
bantime = 1h
findtime = 10m
maxretry = 4
[sshd]
enabled = true
Then sudo systemctl restart fail2ban and sudo fail2ban-client status sshd to see current bans. If you have a fixed IP, put it in the ignoreip setting so a few fat-fingered logins from your own desk can’t ban you.
Automatic security updates and the leftovers
Unpatched software is the dull way servers get breached, far more often than anything exotic. Ubuntu’s unattended-upgrades installs security updates on its own, and the Ubuntu Server documentation on automatic updates covers the setup. On most images it’s already installed. Check with systemctl status unattended-upgrades and look at /var/log/unattended-upgrades/ to confirm it actually ran.
Two things I decide deliberately. First, automatic reboots: the kernel updates need a reboot to take effect, and I enable Unattended-Upgrade::Automatic-Reboot with a 04:00 reboot time only on boxes where a minute of downtime doesn’t matter. On client production servers I leave reboots manual and put it on a monthly calendar reminder. Second, the update scope. The default is security updates only, and I keep it that way. A surprise major-version bump of Postgres at 3am is not a security fix.
Smaller leftovers that take a minute each:
- Remove packages and services you don’t use.
ss -tlnpshows what is listening and which process owns it. - Create a non-root user with sudo, and give it a key before you disable anything.
- Keep
/etc/ssh/sshd_config.d/01-hardening.confand the fail2ban override in version control so the next server is a copy and paste. - Take a backup before you start. I use restic, and my restic backup setup for a small VPS covers the retention and the restore test I keep forgetting to do.
Your next ten minutes
Pick one server you run today, probably the one you set up fastest and have looked at least since. Open a second SSH session first. Then run sudo sshd -T | grep -Ei 'passwordauth|permitroot|maxauth|allowusers' and read what the daemon says, not what you remember. If password authentication is still on, fix that before anything else in this post.
After that, run sudo ufw status verbose and ss -tlnp side by side. Every listening port with no matching reason to be public is your homework for this week. I do this kind of review on client infrastructure as part of my freelance work, and the list it produces is almost always shorter than people fear and more embarrassing than they expect.
