Production Security And Server Hardening

Service-Specific Compromise Playbooks

The general response playbook governs any incident. But the components of a PHP stack fail in characteristic ways, and each has service-specific first moves, evidence locations, and rotation scopes worth writing down before you need them. This lesson is a set of focused annexes to the general plan.

Database Compromise

A breached database is usually the whole ballgame — it holds the data attackers want and the password hashes that unlock accounts elsewhere. First moves: determine scope from the database's own audit and query logs (which is why they must be on and shipped off-host before an incident), and identify what was read, not merely what was changed. Rotate the database credentials and, critically, treat every stored password hash as exposed — force a reset if hashes may have leaked, and be grateful you used a slow algorithm. Personal data access triggers regulatory clocks (GDPR's 72-hour notification), so the legal-notification owner is engaged in parallel, not after. Restore, if needed, must reckon with the attacker possibly having corrupted data slowly over time — the clean backup predates the earliest evidence, not merely yesterday.

Web Application / PHP Compromise

Covered in depth by the application monitoring lesson; the response specifics are: preserve the web root and access logs before touching anything, hunt for persistence beyond the obvious webshell (modified vendor files, injected auth-bypass, added admin accounts, new cron), and assume the application user's every reachable secret is burned. Rebuild the code from version control at a known-good commit rather than cleaning files in place, and diff the restored tree against the integrity manifest to prove it.

Credential / Secret Leak

A leaked secret — committed to a public repo, exposed in a log, phished — is a compromise even with no server intrusion. The clock is brutal because automated scanners find public keys within minutes. First move is rotation, immediately, before investigation, because the exposure is ongoing while the secret is valid. Then assess use: cloud provider keys mean checking for spun-up resources and IAM changes, not just revoking; a leaked API key means auditing what it did while valid. Prevention loops back into the stack: pre-commit secret scanning, short-lived credentials, and secrets managers that make rotation a routine action rather than the emergency this lesson describes.

Dependency / Supply-Chain Compromise

A poisoned package ran with your application's privileges, so the blast radius is the application user's entire reach. First moves: pin to the last known-good version in the lock file, audit what the malicious version could have done (install scripts, runtime egress, secrets in scope), and rotate every credential the application could read — the same scope as an application compromise, because functionally it was one. Then trace how the bad version entered: an unreviewed lock diff, an unpinned constraint, a compromised maintainer account upstream. The controls converge on the committed lock file and the human review of its changes.

Reusing The Pattern

Each annex shares a spine: know where that service's evidence lives and ensure it is shipped off-host in advance; know the first containment move that does not destroy evidence; know the exact rotation scope; and know the recovery source of truth. A new component added to the stack — a message queue, a search cluster, an object store — earns its own annex answering those four questions before it holds anything that matters.

What To Check

Before moving on, make sure you can:

  • give the first moves for a database breach and why hashes count as exposed
  • locate persistence in a compromised PHP application beyond the visible shell
  • explain why a leaked secret is rotated before it is investigated
  • scope rotation for a poisoned dependency and trace its entry
  • state the four questions every service annex must answer

What You Should Be Able To Do

After this lesson, you should be able to extend the general response playbook with service-specific annexes for each component of your stack, each naming that service's evidence, containment, rotation scope, and recovery source before an incident forces the question.

Practice

Practice: Write A Service Annex
Show solution

Evidence: Redis's own logging is thin, so the evidence is mostly external — the firewall/exposure history (was it ever reachable?), the app's connection logs, and INFO/MONITOR output captured at detection. Because that evidence is sparse, the annex leans on prevention: exposure monitoring from the firewall lesson is the primary tripwire, since an exposed Redis is the usual breach path.

First containment move: isolate at the network layer and stop accepting new connections, but snapshot the keyspace first if sessions/queues matter for the investigation — dumping MONITOR briefly can reveal what the attacker touched. Do not simply FLUSHALL; that is evidence destruction dressed as cleanup.

Rotation scope: the Redis requirepass credential; every session token stored there (invalidate all — assume session theft); any secrets cached in Redis (a common and dangerous pattern worth eliminating); and queue payloads, which may contain sensitive data or attacker-injected jobs — drain and inspect rather than trust.

Recovery source: sessions and cache are reconstructable (users re-log in, caches refill), so recovery is a clean Redis instance plus forced re-authentication. Queued jobs are the hard part: replaying a compromised queue can re-execute attacker-injected work, so the recovery decision is to discard in-flight jobs and rely on the application's idempotency and reconciliation rather than replay.

The four questions forced the uncomfortable findings — secrets in cache, unauthenticated exposure as the likely path, and queue-poisoning at recovery — before an incident did.

Practice: Diagnose A Cross-Service Cascade

A leaked deploy key in a public CI log let an attacker read the repository, which contained a committed .env with database and cloud credentials. The team rotated the deploy key and closed the incident. A month later the cloud account showed unexpected charges.

Task

Identify what the response missed and how the service-specific annexes would have scoped it correctly.

Show solution

The response treated a secret leak as a single-credential problem and missed that a leaked secret's rotation scope is everything it exposed, transitively.

The deploy key was the entry, but it granted repository read, and the repository held more secrets — a committed .env. The secret-leak annex says rotate immediately and then assess use; the team did the first for the deploy key and neither for what the deploy key reached. The correct scope was every credential in that repository's history: the database password and the cloud keys, both burned the moment the repo was readable.

The cloud keys specifically needed the credential-leak annex's cloud clause: rotation is necessary but not sufficient, because a cloud key's damage is done by the resources and IAM changes made while it was valid. Revoking it a month late did nothing about the instances the attacker had already spun up — the source of the charges. The annex's first move for cloud keys is check for spun-up resources and IAM changes, not just revoke.

Two prevention loops close it: secrets do not belong in the repository (the .env should never have been committed — pre-commit scanning catches this), and CI logs must not print secrets. But the response failure proper is scope: a secret leak is rotated by exposure reach, and the reach here crossed from CI to repository to database to cloud.

Practice: Define Annex Coverage Checks

Define the checks that keep the set of service annexes complete and current as the stack evolves.

Show solution
  • An inventory job lists every stateful or credential-holding component actually running (databases, caches, queues, object stores, third-party integrations) and diffs it against the set of written annexes; a component with no annex alerts. New infrastructure thus cannot quietly outrun the playbook.
  • Each annex names an evidence source that must be shipped off-host; a freshness monitor on each of those sources doubles as an annex-validity check — an annex pointing at a dead log stream is not executable.
  • Secret-scanning runs on the repository and CI logs, since two of the four annex types (secret leak, dependency) begin with exposure the scanner can pre-empt.

Human, quarterly:

  • For one annex, run its opening moves against staging: can the named person find the evidence, execute the containment step, and enumerate the rotation scope from the written inventory? The annex that cannot be executed on a calm Tuesday will not be executed during an incident.
  • Re-verify each annex's rotation inventory against current reality — every new secret an integration introduces must land in exactly one annex's scope, and the review is what assigns it.
  • Confirm each annex's recovery source (backup, version control commit, reprovisioning) was validated by the general playbook's restore drills.