Production Security And Server Hardening

Reducing Running Services And Attack Surface

Inventory First

You cannot minimize what you have not measured. Three commands give the real picture:

ss -tlnp                       # what is listening, and which process owns it
systemctl list-units --type=service --state=running
systemctl list-timers          # scheduled work counts too

Run these on a server you consider "minimal" and expect surprises: a mail daemon from the base image, an avahi or rpcbind nobody asked for, a database installed for a migration two years ago, a monitoring agent from a departed vendor. Each entry gets a verdict — required, or removed — and "I'm not sure" defaults to removal on a schedule: disable it, wait a bereavement period, then purge.

Disable and remove are different strengths. systemctl disable --now stops a service and prevents its return at boot but leaves the package's code and setuid binaries on disk; apt purge removes them. Prefer purging for anything with no foreseeable return, because code on disk is still surface for privilege escalation even when nothing is running.

The PHP Server's Short List

A typical PHP web server justifies roughly: the web server/proxy, php-fpm, a database if it is co-located, a cache like Redis if used, sshd, a time-sync daemon, the firewall, fail2ban, unattended-upgrades, and the log/metrics shippers the team actually reads. A queue worker or scheduler process if the app uses them. It is a short list on purpose, and each addition should have to argue its way on.

The same discipline applies inside PHP: extensions are attack surface too. Review php -m against what the application's composer.json actually requires, and do not load extensions "just in case." Likewise for the OS package set on container images — smaller base images are this lesson applied to the filesystem.

Keeping It Small

Surface grows back through installs, image updates, and "temporary" experiments. Two habits contain it. First, a committed baseline: the expected services and listeners per server role live in a file, and a periodic job diffs reality against it — the same mechanism as the firewall lesson's exposure checks, because the two reviews share evidence. Second, installation hygiene: Debian and Ubuntu start services automatically upon installation, so installing a package to "try something" means running a daemon until someone notices. Try experiments elsewhere, and let production installs be deliberate.

What To Check

Before moving on, make sure you can:

  • inventory listening sockets, running services, and timers on a host
  • explain the liability bundle a service carries even when idle
  • distinguish disable from purge and choose deliberately
  • defend a minimal service list for a PHP web server role
  • explain how a committed baseline plus diffing keeps surface from regrowing

What You Should Be Able To Do

After this lesson, you should be able to audit a server to a justified-services baseline, remove what fails the audit safely, and keep the baseline enforced so minimalism survives future deploys and installs.

Practice

Practice: Audit A Server To Baseline
Show solution

Verdicts: for each item, answer "what breaks if this stops?" with evidence, not folklore. The app's composer manifest justifies PHP extensions; nginx config justifies upstream services; the runbook justifies agents. Anything justified goes into a committed baseline file with a one-line reason. Anything unjustified is scheduled for removal, and "someone might use it" without a name attached is not a justification.

Safe removal: disable first (systemctl disable --now), watch error rates and logs through a full business cycle including month-end jobs, then purge the package. Removals happen one at a time with a note of what and when, so any breakage bisects trivially. For the truly uncertain — that undocumented daemon a former contractor built — snapshot the server before touching it.

Enforcement: a scheduled job diffs live services and listeners against the baseline file and alerts on drift, so the audit's result becomes a standing property instead of a memory.

Practice: Diagnose Surface Regrowth

A server audited to a clean baseline a year ago now runs a mail daemon (from a package dependency), a node exporter listening on all interfaces (from a "temporary" debugging session), and MySQL (the app migrated to a managed database eight months ago, but the local instance still runs with the old data).

Task

Explain what each regrowth reveals about missing process, and rank the three by risk.

Show solution

Second: the node exporter on all interfaces. It leaks host internals to anyone who can reach the port, and "temporary" instrumentation with no expiry is how debugging tools become permanent surface. Temporary changes need a removal date attached at creation time, and the exporter should have bound to localhost or the monitoring network regardless.

Third: the mail daemon, likely low-risk but emblematic — it arrived silently because package installation starts services by default and nothing diffed the service list afterward.

The shared root cause is that the audit was an event, not a property: without the baseline-diff job alerting on new listeners and services, every mechanism that adds software — dependencies, debugging, migrations — adds surface silently. Restore the diff job first; then handle the three findings in risk order.

Practice: Define Surface Review Checks

Define the checks that keep attack surface minimal on an ongoing basis, covering services, listeners, scheduled work, packages, and PHP extensions.

Show solution
  • Running services diffed against the committed baseline per server role; new or missing services alert with the diff attached.
  • Listening sockets diffed the same way, including bind addresses, shared with the firewall review's evidence.
  • Timers and cron entries diffed against baseline — scheduled work is surface that inventory reviews commonly miss.
  • Count of manually-installed packages tracked over time; steady growth is the smell that prompts the human review early.

Quarterly, human:

  • Re-justify the baseline itself, entry by entry: does the app's current composer.json still require each PHP extension, does each agent still have a reader, does the co-located database still need to be co-located? The baseline must shrink when the architecture does.
  • Check disabled-but-not-purged services past their bereavement period and purge them.
  • For one server, compare a fresh-provisioned instance of the role against the live one; configuration management drift shows up as the difference.