Production Monitoring Orientation

The Three Pillars

Observability rests on three kinds of telemetry, and they answer different questions:

<?php

declare(strict_types=1);

$pillars = [
    'logs'    => 'discrete events with detail: what happened, here, with this context',
    'metrics' => 'aggregated numbers over time: how much, how many, how fast',
    'traces'  => 'the path of one request across functions and services: where the time went',
];

foreach ($pillars as $pillar => $answers) {
    echo $pillar . ': ' . $answers . PHP_EOL;
}

// Prints:
// logs: discrete events with detail: what happened, here, with this context
// metrics: aggregated numbers over time: how much, how many, how fast
// traces: the path of one request across functions and services: where the time went

• Logs are the detailed record. They tell you exactly what happened at a point in time, with context. Their strength is depth; their weakness is that at volume, unstructured logs are a haystack. This is why production logs should be structured (JSON with consistent fields) rather than free-text error_log strings — structured logs are queryable, and queryability is what turns a log pile into an investigation tool.
• Metrics are cheap numbers aggregated over time: request rate, error rate, latency percentiles, queue depth, CPU. They are what dashboards and most alerts are built from, because they are compact enough to keep for a long time and fast to query. They tell you that something is wrong and roughly where, rarely why.
• Traces follow a single request through the code and across service boundaries, timing each span. When a request is slow, a trace shows whether the time went to the database, an external API, or your own code — the question metrics and logs answer only clumsily.

The three are complementary, not competing: metrics alert you that error rate spiked, a trace shows you which dependency, and logs show you the exact failing request. A mature setup lets you pivot between them on a shared identifier (a request or trace ID), which is why propagating that ID everywhere is foundational.

The Golden Signals

You cannot watch everything, so watch the signals that correlate with user pain. Google's SRE practice names four golden signals, and they are a reliable starting dashboard for any service:

• Latency — how long requests take, watched as percentiles (p50, p95, p99), never as an average. Averages hide the tail, and the tail is where users suffer; a p99 of 4 seconds means one in a hundred requests is miserable even if the average looks fine.
• Traffic — how much demand: requests per second, throughput.
• Errors — the rate of failing requests, which requires deciding what "failing" means (5xx obviously; also 4xx spikes, and slow-enough-to-be-useless).
• Saturation — how full the system is: CPU, memory, connection pool, queue depth — the resource that will run out first.

Watching these four per service catches most user-visible problems and gives you the vocabulary to describe them. Everything more specific is added because an incident taught you to.

Instrument For Questions You Will Have

The orientation to carry into the rest of this track: instrumentation is an investment you make before the incident, because you cannot add it during one. The design questions are which signals to emit, how to keep logs structured and correlated, what the golden-signal dashboard for each service looks like, and — crucially — how this ties back to the security monitoring from the hardening track, which is the same discipline pointed at a different class of problem. Detection you cannot act on is noise; the following lessons are about making each signal actionable.

What To Check

Before moving on, make sure you can:

• distinguish monitoring (known signals) from observability (new questions)
• explain what logs, metrics, and traces each answer and where each is weak
• explain why production logs should be structured and correlated by request ID
• name the four golden signals and why latency is watched as percentiles
• explain why instrumentation must precede the incident

What You Should Be Able To Do

After this lesson, you should be able to lay out a monitoring strategy for a PHP service around the three pillars and the golden signals, decide what telemetry to emit, and recognize the difference between data you collect and data you can actually investigate with.