Deployment Success Verification And Watch Windows

A deploy is not done when the new code is running; it is done when you have confirmed the new code is working. Most production incidents are triggered by a change, which means the moments right after a deploy are the highest-risk window in the whole operational cycle — and the one where fast detection pays off most, because the cause is obvious (you just shipped) and the fix is at hand (roll back). This lesson is about closing the loop from "deployed" to "verified," and knowing quickly when to undo.

Verification Is Not "It Deployed"

"The deploy pipeline went green" means the code shipped, not that it works. Real verification checks behavior after the release:

• Smoke tests against production: a handful of automated checks — the health endpoint returns ready, the homepage loads, a critical API responds correctly, a synthetic login succeeds — run immediately post-deploy. These are the health and synthetic checks from the previous lesson, pointed at the freshly deployed release.
• Watch the golden signals for a regression: error rate, latency percentiles, and traffic, compared against the pre-deploy baseline. A deploy that doubles the error rate or the p99 is a failed deploy even if every smoke test passed, because smoke tests sample and metrics see everything.
• Confirm the version actually rolled out: the running release id matches what you shipped, on every instance — partial rollouts and cached old code are real failure modes.

The Watch Window

The watch window is a deliberate period after a deploy — typically 15 to 60 minutes — during which someone (or an automated watcher) actively watches the release's health before declaring success and moving on. It exists because many regressions do not appear instantly: a memory leak takes time to bite, a slow query only hurts under real load, an error only fires on a code path that a user reaches minutes later. Walking away the instant the pipeline goes green means discovering these from customer complaints an hour later instead of from your own dashboard at minute ten.

Practically, the watch window means: deploy, then keep the release's dashboard and error tracker in view (or have an automated canary analysis do it), with the person who shipped staying available to respond. The error tracker's release tagging earns its keep here — a spike of new errors tagged with the new release is the clearest possible "this deploy broke something" signal.

Progressive Delivery Shrinks The Blast Radius

You can make the watch window safer by exposing the new release to fewer users first:

• A canary sends a small fraction of traffic to the new release and compares its golden signals against the old one; if the canary's error rate or latency regresses, it is rolled back having affected only a few percent of users.
• Blue-green runs the new release alongside the old and switches traffic over once verified, keeping the old version warm for an instant rollback.
• Feature flags decouple deploy from release entirely: ship the code dark, then turn the feature on for a small cohort and widen gradually, with an off switch that needs no deploy.

All three serve the same goal — limit how many users a bad release can hurt, and make undoing it fast.

Rollback Is A First-Class Plan

The most important property of a deploy process is that it can be undone quickly. Define, before deploying, what "bad" looks like (error rate or latency crossing a threshold, smoke tests failing, the canary regressing) and what the rollback action is (revert to the previous release, flip the blue-green switch, kill the flag). Rollback should be a rehearsed one-command operation, not an improvisation invented during the incident — the same discipline the compromise playbooks apply to security, applied to deploys. A team that can roll back in two minutes can afford to ship often; a team that cannot treats every deploy as terrifying and ships rarely, which paradoxically makes each deploy bigger and riskier.

What To Check

Before moving on, make sure you can:

• explain why "the pipeline went green" is not deployment verification
• verify a release with smoke tests, golden-signal comparison, and version confirmation
• define a watch window and why regressions are often delayed
• describe canary, blue-green, and feature flags as blast-radius controls
• treat rollback as a pre-defined, rehearsed, one-command plan

What You Should Be Able To Do

After this lesson, you should be able to define what "verified" means for a deploy, run a watch window that catches delayed regressions, use progressive delivery to limit a bad release's reach, and make rollback a fast, rehearsed action rather than a panic.