Production Security And Server Hardening

Compromise Response Playbooks

A playbook is a decision written down before the adrenaline arrives. During an incident, judgment degrades, people improvise, and the improvised instinct — "delete the malicious file and move on" — usually destroys the evidence needed to know how bad it was. A response playbook exists so that the hard choices are made calmly in advance and merely executed under pressure.

The Phases

Incident response has a standard shape, and each phase has a trap the playbook must pre-empt.

  • Detect and declare. Someone has to be able to say "this is an incident" and start the clock. The playbook names who can declare, how they reach the responders, and the principle that a suspected incident is handled as real until disproven — hesitation to declare is the most common early failure.
  • Contain. Stop the bleeding without burning the evidence. Isolate the host at the network layer (a security-group rule to nothing, not a shutdown) so the attacker is cut off but memory and disk survive for analysis. Containment decisions — pull the box, or watch the attacker to learn scope — are genuinely hard and belong in the playbook, not in the moment.
  • Preserve. Snapshot disk and, where possible, memory before changing anything. The off-host logs from the monitoring lesson are the timeline; the snapshot is the crime scene. Both must be captured before eradication, because eradication is destruction.
  • Eradicate and recover. Assume the host is untrustworthy and rebuild it from known-good automation rather than cleaning it in place — you cannot prove you found everything a competent intruder left. Restore data from a backup predating the compromise, which requires knowing when the compromise began, which requires the preserved timeline.
  • Learn. A blameless post-incident review turns one expensive event into cheaper future ones. The output is concrete: the hardening or monitoring gap that let it happen and in, and the change that closes it.

Credential Rotation

Any credential the compromised host could read is burned: database passwords, API keys, tokens, SSH keys, .env secrets, and anything they could reach laterally. Rotation is not optional and not partial — attackers exfiltrate credentials early precisely to survive the rebuild. Rotate before returning the system to service, and let the pain of doing it under pressure motivate the secrets-management that makes rotation a routine operation rather than an archaeology project.

Writing The Playbook

A playbook nobody can execute at 3 a.m. is prose. Keep it concrete: named roles with current contact paths, the exact commands to isolate a host on your provider, where snapshots go, where backups are and how to restore them, the credential-rotation inventory, and who talks to customers and regulators. Store it off the systems it protects — a compromised or down environment cannot serve its own recovery instructions. And rehearse it: a tabletop exercise where the team walks a scenario finds the dead contact, the missing permission, and the backup nobody has actually restored, while those are cheap to fix.

PHP example
<?php

declare(strict_types=1);

$phases = ['detect', 'contain', 'preserve', 'eradicate', 'recover', 'learn'];
foreach ($phases as $i => $phase) {
    echo ($i + 1) . '. ' . $phase . PHP_EOL;
}

// Prints:
// 1. detect
// 2. contain
// 3. preserve
// 4. eradicate
// 5. recover
// 6. learn

What To Check

Before moving on, make sure you can:

  • name the response phases and the trap each one pre-empts
  • explain why containment must preserve evidence and how isolation differs from shutdown
  • justify rebuild-over-clean and backup-restore-predating-compromise
  • scope credential rotation to everything the host could reach
  • state what makes a playbook executable rather than aspirational

What You Should Be Able To Do

After this lesson, you should be able to write a response playbook your team could actually run, make the hard containment and recovery decisions in advance, scope credential rotation correctly, and rehearse the plan so its gaps surface before an incident does.

Practice

Practice: Write A Response Playbook
Show solution

Contain: isolate the affected host by replacing its security group with a deny-all (keeping the responder's admin path), never by shutdown — memory and disk are evidence. The pre-made decision: isolate immediately rather than observe, because a small team cannot safely watch a live intruder.

Preserve: snapshot the disk (and memory image if the provider supports it) before any change; confirm the off-host logs cover the window. Both are captured before eradication, always.

Eradicate/recover: rebuild the host from provisioning automation, never clean in place. Restore data from the latest backup predating the earliest compromise evidence — which is why the timeline came first. Validate the rebuild against the service baselines before returning traffic.

Rotate: a written inventory — database passwords, all API keys, third-party tokens, SSH keys, .env secrets, session/signing keys — rotated before the service is public again. Sessions and keys are invalidated with the one-command tools from the application-monitoring lesson.

Communicate: named owner for customer and regulatory notification, with a decision tree for disclosure obligations.

Storage and rehearsal: the playbook lives in a separate repo and a printed copy, off the production account, and is tabletop-tested quarterly.

Practice: Diagnose A Botched Response

Task

Identify each response error and the phase it violated.

Show solution
  • Preserve, violated first and worst: deleting the webshell and restarting destroyed the disk state, the running processes, and the memory that would have shown how the attacker got in and what else they touched. The restart also aged out local logs. There is now no way to scope the incident — the single most expensive mistake, because it forecloses every later answer.
  • Contain, misunderstood: a restart is not containment. The attacker's access route — the upload vulnerability and whatever persistence they added beyond the one visible shell — survived untouched, which is why they walked back in.
  • Eradicate, done by cleaning instead of rebuilding: removing one artifact assumes it was the only one. A competent intruder leaves several; rebuild-from-known-good is the only way to assert the host is clean, and it was skipped.
  • Recover/rotate, absent: the database password the host could read was never rotated, so even a full rebuild would have handed the attacker a valid credential. Rotation scopes to everything the host could reach.
  • Learn, impossible: with no evidence, no post-incident review could identify the entry vulnerability, so it stayed open — the direct cause of the recurrence.

The meta-error is having no playbook: each destructive step was a reasonable-seeming improvisation, which is exactly what a written, rehearsed plan replaces.

Practice: Define Playbook Readiness Checks
Show solution
  • Backups are restore-tested on a schedule, not just taken; a backup never restored is a hypothesis. The restore drill records how long it took and whether the data was intact.
  • Off-host log retention covers at least the detection-lag the monitoring lesson assumes; a retention shorter than your realistic time-to-detect means evidence expires before you look.
  • Contact-path liveness: the paging rota resolves to current employees, checked whenever anyone joins or leaves.

Human, quarterly:

  • A tabletop exercise walks one scenario end to end against the written playbook. The findings are always concrete: a permission the responder lacks, a snapshot command that changed, a credential missing from the rotation inventory. Fix them while cheap.
  • The credential-rotation inventory is diffed against reality — every new integration adds a secret, and the inventory that misses one leaves the attacker a key after recovery.
  • The playbook's off-system copy is confirmed reachable without the production account, because the one time you need it is the one time that account may be gone.

The single most revealing check is the restore drill: teams consistently discover their backups, response plan, or both do not survive contact with an actual recovery.