Observability Monitoring And Incident Response

Health Pings, Uptime, And Synthetic Checks

Health Check Endpoints

A health check is an endpoint the application exposes so infrastructure can ask "are you ready to serve?" The critical distinction is between two kinds, because conflating them causes outages:

  • A liveness check answers "is the process alive?" — cheap, no dependencies, so an orchestrator knows whether to restart the process.
  • A readiness (or deep) check answers "can I actually serve a request right now?" — it verifies the dependencies the app needs: database reachable, cache reachable, disk writable. A load balancer uses this to decide whether to send traffic to an instance.
PHP example
<?php

declare(strict_types=1);

function readiness(callable $dbOk, callable $cacheOk): array
{
    $checks = ['database' => $dbOk(), 'cache' => $cacheOk()];
    $healthy = !in_array(false, $checks, true);
    return ['status' => $healthy ? 'ok' : 'degraded', 'checks' => $checks];
}

$result = readiness(fn () => true, fn () => true);
echo $result['status'] . PHP_EOL;
http_response_code($result['status'] === 'ok' ? 200 : 503);

// Prints:
// ok

Two design cautions. A readiness check must return a proper status code — 200 healthy, 503 not — because that code is the signal the load balancer acts on; a check that always returns 200 regardless of dependency state is worse than none, because it hides outages. And a deep health check is itself a small attack and load surface: keep it unauthenticated-but-minimal, do not leak internal detail (versions, connection strings) in its body, and be aware that an expensive readiness check hit frequently by many probers can itself cause load.

Uptime And Synthetic Monitoring

Uptime monitoring is an external service that pings your health endpoint (or homepage) from outside your infrastructure on an interval and alerts when it fails. Because it runs outside, it catches the whole class of failures internal monitoring cannot see: expired TLS certificates, DNS misconfiguration, network/routing failures, an entire-region outage, and a load balancer that is up but pointing nowhere. Probing from multiple geographic locations distinguishes "the site is down" from "the site is unreachable from one region," and avoids paging on a single prober's local network hiccup.

Synthetic monitoring goes further: instead of pinging one endpoint, it scripts a real user journey — load the login page, sign in, add to cart, check out — and runs it continuously against production. This catches broken functionality that is invisible to health checks and metrics: the checkout that returns 200 but silently fails to charge, the login that works but redirects to an error, the third-party widget that broke the page. Synthetic checks watch the critical business flows the same way a user would, and they often catch breakage before real users hit it. The trade is maintenance — scripts break when the UI changes — so reserve them for the handful of journeys that actually matter (login, purchase, signup) rather than every page.

Where This Fits

Health checks feed the orchestrator and load balancer; uptime and synthetic checks feed alerting from the user's vantage point and, as the next lesson shows, are exactly the probes you watch during a deploy to confirm a release is actually serving. Together with internal metrics and error tracking, they close the gap between "our dashboards are green" and "our users can use the product."

What To Check

Before moving on, make sure you can:

  • distinguish liveness from readiness and say what each consumer does with it
  • return correct status codes and avoid the always-200 health check
  • keep health endpoints minimal and non-leaky as an attack/load surface
  • explain what external uptime monitoring catches that internal monitoring cannot
  • describe synthetic monitoring of critical journeys and its maintenance trade-off

What You Should Be Able To Do

After this lesson, you should be able to build liveness and readiness endpoints that infrastructure can trust, set up external uptime monitoring that catches the failures your internal view misses, and script synthetic checks for the business-critical journeys that matter most.

Practice

Practice: Design Health And Uptime Monitoring
Show solution

Readiness (/health/ready): checks the dependencies needed to serve — database connection, Redis, and writable storage — and returns 200 only if all pass, 503 otherwise. The load balancer uses this to pull an unhealthy instance out of rotation. The body lists per-dependency status for humans but leaks no versions or connection strings, and the check is lightweight enough to survive frequent probing.

External uptime: a third-party service probes the homepage and /health/ready every minute from several regions, alerting on failure. Multi-region distinguishes a real outage from one prober's local hiccup and catches TLS expiry, DNS, and routing failures the internal dashboards cannot see. TLS-expiry alerting is included explicitly (14 days out), tying to the reverse-proxy lesson.

Synthetic journeys — only the ones that matter: (1) log in, (2) search and add to cart, (3) complete a test checkout against a sandbox payment path. These run continuously and catch "200 but broken" failures like a checkout that stops charging. Non-critical pages are left to metrics and error tracking, keeping synthetic-script maintenance bounded to the flows whose breakage costs money.

Practice: Diagnose A Silent Outage

Task

Explain why the monitoring missed both and what checks would have caught them.

Show solution

Both failures live in the gap between "the servers are healthy" and "users can use the product," which internal monitoring structurally cannot see.

Expired TLS certificate: from inside, every service was fine — the app ran, the database answered, metrics were green — because the failure was at the TLS layer between the user and the load balancer, outside the internal vantage point entirely. Browsers refused the connection while dashboards saw nothing wrong. Only an external probe experiences what the user experiences. Fix: external uptime monitoring from multiple regions hitting the real HTTPS endpoint, plus explicit certificate-expiry alerting well before the 90-day cliff. The first alert should have come from a prober days earlier (expiry warning), not from users at the moment of failure.

Silent checkout failure: the health check returned 200 because the homepage loaded, but "homepage loads" says nothing about whether checkout charges cards. A shallow health check confirms the process is up, not that business functions work — the always-green trap. Fix: a synthetic check that scripts the actual purchase journey against a sandbox payment path and runs continuously. It would have failed the moment charging broke and paged within minutes instead of a day, because it exercises the function a user cares about rather than a proxy for it.

The unifying lesson: internal metrics, external uptime, and synthetic journeys are three different vantage points, and the last two catch exactly the failures the first is blind to. A product is only "up" if a user, from outside, can complete the journey that matters.

Practice: Define Health/Uptime Review Checks

Define the checks that keep health endpoints and external monitoring trustworthy.

Show solution
  • A test forces a dependency down (e.g. point the app at an unreachable database in a test env) and asserts readiness returns 503, not 200 — directly guarding against the always-200 regression that hides outages.
  • A test asserts liveness stays 200 when a dependency is down (so a dead DB does not trigger pointless restarts) — the liveness/readiness distinction, enforced.
  • External uptime and TLS-expiry monitors are themselves monitored for freshness; a dead prober looks like an up site otherwise.
  • Synthetic journeys run on schedule and their pass/fail is alerted; a synthetic that has been silently failing (or disabled) is flagged.

Human, periodically:

  • Confirm the synthetic scripts still match the current UI and still cover the business-critical journeys — UI changes break them, and a stale synthetic gives false confidence.
  • Review what external monitoring covers against recent incident types; every "how did we not catch that?" outage should add or adjust an external/synthetic check.
  • Verify health endpoints still leak no sensitive internal detail as the app evolves.