Datadog-Style Infrastructure And Service Monitoring

Error tracking watches your code; infrastructure and service monitoring watches everything the code runs on and depends on — hosts, containers, databases, queues, and the request paths through them. Datadog is a common commercial example, and the open-source Prometheus-and-Grafana stack does the same job; "Datadog-style" here means the whole category of metrics-collection, dashboarding, and alerting platforms. This lesson is about turning the golden signals from the orientation into a living system of dashboards and alerts.

Collecting Metrics

Metrics come from two directions. Infrastructure metrics — CPU, memory, disk, network, and per-service resource use — are gathered by an agent running on each host (the Datadog agent, or Prometheus node/exporter processes). Application metrics — request rate, latency percentiles, business counters like signups or orders — are emitted by your code. There are two collection models worth knowing: push, where the app sends metrics to a collector (StatsD-style, Datadog), and pull/scrape, where the platform periodically fetches a metrics endpoint your app exposes (Prometheus). PHP's request-per-process model makes the scrape model slightly awkward — a short-lived request cannot hold counters in memory between requests — so PHP apps commonly push to a local agent or write to a shared store the exporter reads. The mechanism matters less than the discipline of emitting the right metrics.

Two vocabulary points that prevent mistakes. Metrics have types: counters (monotonic totals like requests served), gauges (a value that goes up and down like queue depth), and histograms (distributions, which is how latency percentiles are computed). And metrics carry tags/labels (endpoint, status, region) that let you slice one metric many ways — but every distinct tag combination is a separate stored series, so putting a high-cardinality value like a user id or a raw URL in a tag explodes storage and cost. This is the exact same cardinality warning as the security-monitoring lesson: use bounded labels (route templates, not filled-in paths).

Dashboards That Answer Questions

A good dashboard is designed around the questions you ask under stress, not around every metric you can plot. The golden signals per service go up top; dependencies (database, cache, queue, external APIs) below; saturation of the resources most likely to run out (connection pools, queue depth, disk) alongside. The test of a dashboard is whether, during an incident, it answers "is it us or a dependency, and what is saturating?" in one glance. Sprawling walls of forty graphs fail that test — they are collected, not designed. Keep a small number of purpose-built dashboards over a museum of everything.

Alerts Worth Waking For

An alert is a promise to interrupt a human, so the bar is high: every alert should be actionable (there is something to do), urgent (it cannot wait for business hours if it pages), and real (low false-positive rate). The failure mode this prevents is alert fatigue — when a channel cries wolf often enough, humans start ignoring it, and the one real alert is missed inside the noise. This is not hypothetical; it is the most common way monitoring fails in practice, and you saw its shape in the security track's dead-inbox incident.

Alerting design principles that keep the bar high:

• Alert on symptoms, not causes. Page on "checkout error rate above 2%" (user pain) rather than "CPU at 90%" (which may be perfectly fine under load). High CPU that is not hurting anyone is not an emergency; a failing checkout is.
• Use severity tiers. Page (wake someone) for user-facing outages; notify (a channel message) for degradations to look at soon; ticket for slow trends. Not everything deserves a 3 a.m. phone call.
• Set thresholds from data, with duration. A threshold derived from normal behavior, requiring the condition to persist for a few minutes, filters transient blips. "Error rate above 2% for 5 minutes" beats "any single failed request."
• Every paging alert has a runbook. If there is nothing to do, it should not page — and if there is, the runbook lesson says write it down.

<?php

declare(strict_types=1);

$alerts = [
    ['symptom' => 'checkout 5xx > 2% for 5m', 'severity' => 'page'],
    ['symptom' => 'p99 latency > 2s for 10m', 'severity' => 'page'],
    ['symptom' => 'queue depth rising 30m',   'severity' => 'notify'],
    ['symptom' => 'disk > 80%',               'severity' => 'ticket'],
];

foreach ($alerts as $a) {
    echo str_pad($a['severity'], 7) . ' <- ' . $a['symptom'] . PHP_EOL;
}

// Prints:
// page    <- checkout 5xx > 2% for 5m
// page    <- p99 latency > 2s for 10m
// notify  <- queue depth rising 30m
// ticket  <- disk > 80%

SLIs, SLOs, And Error Budgets

The mature framing ties monitoring to a reliability target. A service level indicator (SLI) is a measured signal (say, the fraction of requests served under 300 ms and without error). A service level objective (SLO) is the target for it (99.9% over 30 days). The gap between the SLO and 100% is the error budget — the amount of unreliability you are allowed to spend. This reframes alerting productively: page when the error budget is burning fast, not on every blip, and use the remaining budget to decide whether to ship features or stabilize. It turns "how reliable is reliable enough?" from an argument into a number.

What To Check

Before moving on, make sure you can:

• distinguish infrastructure from application metrics and push from scrape collection
• name metric types (counter, gauge, histogram) and the cardinality trap in tags
• design a dashboard that answers "us or dependency, and what is saturating?"
• state the actionable/urgent/real bar and what alert fatigue costs
• alert on symptoms not causes, with severity tiers and data-derived thresholds
• define SLI, SLO, and error budget and how they reframe alerting

What You Should Be Able To Do

After this lesson, you should be able to instrument a PHP service's infrastructure and application metrics with bounded cardinality, build purpose-designed dashboards, and write a small set of symptom-based, tiered alerts that stay actionable instead of decaying into fatigue.