Production Security And Server Hardening

SSH Hardening And Emergency Access

The Core Hardening Moves

Key-based authentication replaces guessable passwords with cryptographic keys. Generate a modern key (ssh-keygen -t ed25519), install the public key on the server, confirm login works, and only then disable password authentication.

# /etc/ssh/sshd_config
PasswordAuthentication no
PermitRootLogin no
KbdInteractiveAuthentication no

PermitRootLogin no forces attackers to guess both a username and a key, and forces operators through a personal account and sudo, which leaves an audit trail. Apply changes with a reload (systemctl reload ssh) — a reload does not drop existing connections, which matters for the safety ritual below.

Fail2ban (or the distribution's equivalent) watches the auth log and bans IPs that fail repeatedly. It does not make a key-only server meaningfully safer, but it keeps logs readable and slows credential-stuffing against any service it watches. Moving SSH to a non-standard port only reduces log noise; it is not a security control and should never substitute for the settings above.

The Lockout Ritual

Every SSH change follows the same discipline: change the config in one session while a second, already-authenticated session stays open. Test a fresh login from a third terminal. Only when the fresh login succeeds do you close anything. Operators skip this exactly once.

Emergency Access

Key-only SSH plus a firewall means a lost key or a bad config can lock you out of your own server. Decide the recovery path before you need it:

  • The provider console. Every serious VPS and cloud provider offers out-of-band console access that bypasses SSH entirely. Confirm it works and that the root or console password needed for it exists somewhere retrievable.
  • A break-glass key. A second authorized key whose private half is stored offline — printed, or in a sealed password-manager entry with team access rules — covers the lost-laptop case.
  • Documentation. The recovery steps live in the runbook, not in the head of the person who set them up.

Emergency access is itself an attack surface: audit who can reach the provider console with the same seriousness you audit SSH keys, because it is a superset of SSH.

Key Hygiene

Authorized keys accumulate. Review ~/.ssh/authorized_keys on every account periodically, remove keys belonging to departed people and decommissioned machines, and prefer one key per person per device so removal is surgical. When a laptop is lost or a teammate leaves, key removal is a compromise-response step with a deadline, not a cleanup chore.

What To Check

Before moving on, make sure you can:

  • explain why key-based auth plus disabled passwords defeats bulk SSH attacks
  • state what PermitRootLogin no changes about both attacks and audit trails
  • describe the second-session ritual and why reload beats restart during it
  • name two emergency-access paths that survive a locked-down SSH
  • explain why a non-standard port is noise reduction, not security

What You Should Be Able To Do

After this lesson, you should be able to take a fresh server from password SSH to key-only access without risking lockout, set up and verify an emergency path, and run periodic key reviews as part of normal operations.

Practice

Practice: Plan An SSH Lockdown

You have one fresh Ubuntu VPS reachable only by root password over SSH. Write the ordered steps to reach key-only, non-root SSH with a verified emergency path, marking the point in the sequence where each risky change is verified.

Show solution
  1. Confirm the provider console works with the root password — the emergency path is verified before anything can go wrong, not after.
  2. Create a personal account, add it to sudo, and set a strong password for sudo use.
  3. Generate an ed25519 key locally; install the public key for the personal account.
  4. From a new terminal, verify key login as the personal user and that sudo works. Keep this session open.
  5. Edit sshd_config: PermitRootLogin no, PasswordAuthentication no. Reload, do not restart.
  6. From a third terminal, verify a fresh key login succeeds and a password attempt is refused.
  7. Register the break-glass key: a second keypair stored offline, its public half added, its login tested once.
  8. Only now close the standing sessions, then document the recovery paths in the runbook.

The two verifications that must not move later in the sequence: the console test (step 1) before any change, and the fresh-login test (step 6) before closing the safety session. Every lockout story skips one of those two.

Practice: Diagnose A Lockout Near-Miss

Task

List each broken practice and what the correct discipline was.

Show solution
  • No fresh-login test. The typo would have surfaced in thirty seconds if a new connection had been tested while an existing session stayed open. Closing the laptop without that test converted a typo into a lockout.
  • Restart instead of reload, and no standing session. systemctl reload keeps existing connections alive precisely so a bad config leaves you a working session to fix it from. A restart plus a closed laptop left zero paths in.
  • Emergency access assumed, not verified and not owned. The console password existing "somewhere" is not an emergency path; it must be tested periodically and stored where the current team can retrieve it. Departure offboarding should have caught the password manager, which makes this a key-hygiene failure too.

Recovery now goes through the provider's password-reset or rescue-boot process — slower, and itself worth documenting once survived.

Practice: Define SSH Review Checks
Show solution
  • sshd_config effective values (via sshd -T) still show permitrootlogin no and passwordauthentication no on every host — drift here is silent and serious.
  • Every key in every authorized_keys maps to a current person and device; keys for departed people or retired laptops are removed on the spot.
  • The break-glass key's offline copy is where the runbook says it is, and one member logs in with it to prove it still works.
  • The provider console is reachable by at least two current team members, and its credential was rotated since the last departure.
  • Fail2ban (or equivalent) is running and has recent bans, which doubles as proof it is reading the right log.
  • Auth logs are shipped off-host and someone can produce last month's successful-login list per server in under five minutes.

The recurring theme: every path in — normal, break-glass, and console — is tested, owned, and mapped to current humans.