Overlapping Jobs, Locks, And Leases

Cron does not check whether the previous run of a job has finished before starting the next one. A job scheduled every minute that occasionally takes 90 seconds will, sooner or later, have two instances running at once — reading and writing the same data, sending the same notifications twice, or racing each other in ways that are hard to reproduce because they depend on exact timing. This lesson covers preventing overlap with locks, why a plain lock isn't quite enough for distributed systems (which is where "lease" comes in), and fencing tokens for the failure mode leases alone don't cover.

Why Overlap Happens

A job's runtime is not fixed — it depends on data volume, network latency to a dependency, database load, and a dozen other things that vary run to run. A job that takes 20 seconds on a normal day can take 3 minutes when a downstream API is slow or the table it scans has grown. If that job is scheduled every minute, cron doesn't know or care that the previous instance is still running — it starts a new one on schedule regardless. Now two instances are executing the same logic against the same data concurrently, and whatever that logic assumed about running alone (a "read current balance, then update it" sequence, for example) can produce wrong results neither instance would produce alone.

File Locks: The Local Solution

For a job running on a single server, a file lock is the simplest fix — acquire an exclusive, non-blocking lock at the start; if you can't get it, another instance is already running, so exit immediately instead of waiting or duplicating work.

<?php

declare(strict_types=1);

function tryRunExclusively(string $lockFile, callable $job): bool
{
    $handle = fopen($lockFile, 'c');
    if ($handle === false) {
        throw new RuntimeException("Cannot open lock file: {$lockFile}");
    }

    if (!flock($handle, LOCK_EX | LOCK_NB)) {
        fclose($handle);
        return false; // another instance already holds the lock
    }

    try {
        $job();
        return true;
    } finally {
        flock($handle, LOCK_UN);
        fclose($handle);
    }
}

$lockFile = tempnam(sys_get_temp_dir(), 'cron-lock-');

// Simulate a first instance holding the lock open for the whole "run"
$firstHandle = fopen($lockFile, 'c');
flock($firstHandle, LOCK_EX);

// A second instance tries to start while the first is still running
$secondRan = tryRunExclusively($lockFile, function () {
    echo "second instance ran\n";
});
var_dump($secondRan);

// The first instance finishes and releases its lock
flock($firstHandle, LOCK_UN);
fclose($firstHandle);

// Now a third attempt succeeds
$thirdRan = tryRunExclusively($lockFile, function () {
    echo "third instance ran\n";
});
var_dump($thirdRan);
// Prints:
// bool(false)
// third instance ran
// bool(true)

LOCK_EX | LOCK_NB requests an exclusive lock without blocking — if another process already holds it, flock() returns false immediately instead of waiting, which is exactly what a scheduled job wants: skip this run rather than queue up behind a slow one, which would only make the overlap worse as delayed runs pile up.

A key property of OS-level file locks: if the process holding one crashes or is killed, the operating system releases the lock automatically when the process's file descriptors are cleaned up. This means a plain flock() doesn't have the "stale lock held forever by a dead process" problem — which is not true of the distributed locks covered next, and is exactly why they need an extra idea.

Why Distributed Locks Need An Expiry: Leases

A file lock only protects a single machine. Once a job runs on multiple servers — several web servers each with the same crontab, or a horizontally-scaled worker fleet — you need a lock that all of them can see, typically implemented with a shared store like Redis or a database.

The problem: unlike an OS file lock, a distributed lock isn't automatically released just because the process that acquired it crashed — nothing about Redis knows or cares that the PHP process on server A that called SET lock:daily-report ... NX just died. Without an expiry, a crashed holder leaves the lock held forever, and every future run is blocked indefinitely by a lock nobody will ever release.

The fix is a lease: a lock with a built-in expiry (TTL), so even if the holder crashes without releasing it, the lock frees itself automatically after the TTL passes.

<?php

declare(strict_types=1);

// no-execute: requires a live Redis server via the redis extension.
$redis = new Redis();
$redis->connect('127.0.0.1', 6379);

$token = bin2hex(random_bytes(16));
$acquired = $redis->set('lock:daily-report', $token, ['NX', 'EX' => 30]);

if (!$acquired) {
    exit(0); // another node already holds the lease
}

try {
    // ... run the job; must finish comfortably within 30 seconds ...
} finally {
    $release = <<<'LUA'
    if redis.call("GET", KEYS[1]) == ARGV[1] then
        return redis.call("DEL", KEYS[1])
    end
    return 0
    LUA;
    $redis->eval($release, ['lock:daily-report', $token], 1);
}

NX means "set only if the key doesn't already exist" (the acquire step), and EX 30 sets a 30-second expiry — the lease. If this process dies mid-job, Redis expires the key on its own after 30 seconds, and the next scheduled attempt can acquire it. The TTL must be chosen deliberately: too short, and a legitimately-still-running job can lose its lease to another instance while it's mid-work; too long, and a genuine crash leaves the resource blocked for longer than necessary.

The Release Must Check Ownership: Why The Token Exists

Notice the release step doesn't just DEL the key — it checks that the stored value still matches the unique $token this specific process generated when it acquired the lease. This matters because of a specific race: if a job runs slightly longer than its TTL, the lease can expire and be acquired by a second instance while the first instance is still finishing up. If the first instance's cleanup code then blindly ran DEL lock:daily-report, it would delete the second instance's currently-active lease — not its own, which already expired — leaving the resource unprotected while the second instance still believes it holds an exclusive lease. Comparing the token before deleting (done atomically via the Lua script, so the check-then-delete can't itself race) ensures a process only ever releases a lease it still actually owns.

Fencing Tokens: The Failure The Token Check Alone Doesn't Solve

The token check prevents deleting someone else's lease, but there's a subtler problem it doesn't solve: what if the first instance, after its lease has already expired and been reacquired by a second instance, is still slow enough to make one more write to the actual protected resource (the database, the report file) before it notices its lease is gone? Both instances can end up writing to the shared resource around the same time, even though only one of them currently "holds the lease" — the lease only ever protected the right to start, not a guarantee that the protected write always happens while still valid.

The fix is a fencing token: a number that increases every time the lease is acquired (e.g., a Redis INCR alongside the lock), passed along with every write to the protected resource itself. The resource (a database update, a storage API) is told to reject any write carrying a fencing token lower than the highest one it's already seen. If instance A acquired the lease with token 7, lost it, and instance B acquired it next with token 8, any late write from A still carrying token 7 is rejected by the resource — even though A doesn't know it's no longer the legitimate holder, the resource does. This requires the protected resource itself to understand and check fencing tokens, which is more machinery than most scheduled PHP jobs need — reserve it for the cases where a stale write actually causes serious harm (financial transactions, anything non-idempotent), not as a default for every locked job.

Failure Modes

The most common failure is not locking at all, on the assumption that overlap "probably won't happen" — it will, the moment the job's runtime occasionally exceeds its interval, which for most real workloads is eventually.

The second is a distributed lock with no expiry, which converts a single crashed process into a permanently stuck job — every future scheduled run blocked by a lock nobody will ever release, discovered only when someone notices the job hasn't run in days.

The third is releasing a lease without checking ownership, which can silently delete a different instance's active lease during exactly the overlap scenario the lease was meant to prevent.

What To Check

Before moving on, make sure you can:

• explain why a job's runtime occasionally exceeding its schedule interval makes overlap inevitable, not just theoretically possible
• implement a non-blocking flock()-based exclusive run for a single-server scheduled job
• explain why OS file locks release automatically on crash, and why distributed locks don't
• implement a lease (lock + TTL) and explain what TTL value tradeoff it represents
• explain why a lease's release step must check ownership (via a token) before deleting
• describe what a fencing token adds beyond a token-checked release, and why it requires cooperation from the protected resource itself

What You Should Be Able To Do

After this lesson, you should be able to add overlap protection to a scheduled PHP job: a non-blocking file lock for single-server jobs, or a TTL'd, token-verified lease for distributed ones — and reason about when the added complexity of fencing tokens is actually warranted versus unnecessary for a given job's risk profile.