Production Security And Server Hardening

Updates, Patching, And Vulnerability Windows

Two Streams, Two Policies

Security updates and feature updates deserve different treatment. Security fixes should flow automatically: on Debian and Ubuntu, unattended-upgrades applies the security pocket on a daily timer, and the risk calculus is lopsided — the harm from a rare bad security patch is dwarfed by the harm from being the server that stayed vulnerable for six weeks. Configure it, then verify it is actually running; a surprising number of servers have the package installed with the timer dead.

apt list --upgradable            # what is waiting
/var/log/unattended-upgrades/    # what auto-applied, and when
cat /var/run/reboot-required     # kernel or core library wants a restart

Feature and major-version updates stay deliberate: staged through a non-production environment, scheduled, with a rollback plan. The mistake to avoid is letting deliberateness leak into the security stream — "we apply everything monthly in a maintenance window" means a median two-week vulnerability window by policy.

Some patches only take effect on restart: kernels need a reboot (or live-patching), and a patched OpenSSL is not protecting a php-fpm process that loaded the old library at startup. reboot-required and needrestart exist because "patched on disk, vulnerable in memory" is a real state. Rebooting must therefore be an ordinary, rehearsed event — the zero-downtime patterns in the deployment track are what make that cheap.

The Application Is Also A Surface

The OS package manager does not see Composer dependencies. composer audit checks the lock file against known-vulnerability databases and belongs in CI and on a schedule, because a vulnerability can be published against a dependency you deployed months ago — the alert has to find you, not wait for your next deploy. The same applies to the PHP runtime itself: track your version's support status, since a PHP version past security-support means every future CVE is permanent.

Subscribe to security announcements for the handful of components that matter most — the distribution, PHP, and your framework — so the rare drop-everything patch (the kind that arrives with a logo and a deadline) is something you hear about within hours.

Knowing Your Windows

Measure the window honestly: when a relevant CVE lands, how long until production runs the fix? For auto-applied OS security patches the answer should be about a day. For Composer dependencies, the length of your audit-alert-deploy loop. For anything requiring a reboot, add the time to your next rehearsed restart. If any of those numbers embarrasses you, that — not more scanning — is the thing to fix.

What To Check

Before moving on, make sure you can:

  • define a vulnerability window and why it is the real metric
  • justify automatic security patching against manual batching
  • explain patched-on-disk versus patched-in-memory and the reboot discipline
  • place composer audit in both CI and scheduled monitoring
  • name the announcement channels worth subscribing to

What You Should Be Able To Do

After this lesson, you should be able to configure and verify automatic security updates, keep application dependencies under continuous audit, handle reboot-requiring patches without drama, and state your actual vulnerability windows with evidence.

Practice

Practice: Design A Patching Policy
Show solution

Reboots: reboot-required is checked by monitoring; servers reboot within a week of the flag, one at a time, using the load balancer drain the team already rehearses monthly. Kernel patches thus have a bounded in-memory window without anyone being brave.

Feature updates: applied monthly to the staging server first, soaked for a week, then rolled out. Major version upgrades (new PHP minor, new Ubuntu LTS) get their own planned change with rollback via snapshot.

Application: composer audit blocks CI on known vulnerabilities and also runs nightly against the deployed lock file, alerting the team chat — the nightly run is what catches CVEs published after the last deploy. Framework and PHP security lists are subscribed to a shared inbox with a same-day triage rule.

The policy fits on one page, and every clause has a verification, because a patching policy without evidence is a wish.

Practice: Diagnose A Long Window

A breach investigation finds the entry point was a PHP framework vulnerability patched upstream 41 days earlier. The team had unattended-upgrades working, CI running composer audit on every deploy — and no deploys in the six weeks before the breach.

Task

Explain how each defense failed to cover this case and what closes the gap.

Show solution

Each defense was real but scoped away from this vulnerability.

Unattended-upgrades covers distribution packages; a Composer-installed framework is invisible to apt, so the OS layer was never in play.

composer audit in CI answers "is it safe to deploy this lock file now?" — a gate, not a monitor. With no deploys for six weeks, the gate never ran, and a vulnerability disclosed after the last deploy aged unexamined in production. The audit passed the day it deployed and was never asked again.

The gap is monitoring the deployed artifact on a clock: a nightly composer audit against the production lock file (or a dependency-alerting service watching the repository) alerting to a channel with a same-day triage rule. That converts "days until next deploy" into "hours until a human knows."

Two supporting fixes: subscribe to the framework's security list, because critical framework CVEs are announced loudly and 41 days of silence means nobody was listening; and treat a security-fix release as its own deploy trigger — teams that only deploy features inherit feature cadence as their patch cadence, which is exactly what happened here.

Practice: Define Patching Review Checks

Define the checks that prove a patching policy is working, covering OS, reboots, application dependencies, and the vulnerability-window metric itself.

Show solution
  • Every server's unattended-upgrades log shows a run within 24 hours, and no security update is pending older than 48 hours; either condition failing alerts.
  • reboot-required age per server is graphed; anything over seven days pages.
  • Nightly composer audit of the deployed lock file is green, or its finding has an open ticket with an owner.

Monthly, human:

  • Sample one recent CVE relevant to the stack and reconstruct its actual window from logs: disclosure date, detection date, fix-in-production date. Publishing that number to the team is the review — it keeps the metric honest and the process improving.
  • Confirm PHP and framework versions remain inside security support, and that end-of-life dates for both appear on the team calendar with migration lead time.
  • Verify the security announcement inbox was triaged (no unread items older than a day), because subscriptions rot quietly.