General API Idempotency And Retry Design

Idempotency means the same operation can be repeated without creating a second side effect. In API work, it is what allows clients to retry after a timeout without accidentally creating two orders, two payments, two tickets, or two emails.

Retries and idempotency belong together. A retry policy answers "when should the client try again?" Idempotency answers "what happens if the first attempt actually worked but the client never saw the response?"

Natural idempotency

Some HTTP methods are normally idempotent by design:

• GET /users/42 can be called repeatedly and should return the same resource state unless something else changes it.
• PUT /users/42 replaces a known resource, so repeating the same request should leave the resource in the same state.
• DELETE /users/42 should leave the resource deleted even if called twice.

POST is usually not naturally idempotent because it often creates a new server-chosen resource.

<?php

declare(strict_types=1);

function naturallyIdempotent(string $method): bool
{
    return in_array(strtoupper($method), ['GET', 'HEAD', 'PUT', 'DELETE'], true);
}

foreach (['GET', 'POST', 'PUT', 'DELETE'] as $method) {
    echo $method . ': ' . (naturallyIdempotent($method) ? 'yes' : 'no') . PHP_EOL;
}

// Prints:
// GET: yes
// POST: no
// PUT: yes
// DELETE: yes

That does not mean every real implementation is perfect. For example, a DELETE endpoint that sends a cancellation email every time it is called is not idempotent in practice.

Idempotency keys

For create-style POST operations, clients can send an idempotency key. The server stores the key with the first response. If the same key arrives again, the server returns the stored response instead of running the operation again.

<?php

declare(strict_types=1);

function createPayment(array $request, array &$idempotencyStore): array
{
    $key = trim((string) ($request['idempotency_key'] ?? ''));

    if ($key === '') {
        return ['status' => 400, 'body' => ['error' => 'idempotency_key is required']];
    }

    if (isset($idempotencyStore[$key])) {
        return $idempotencyStore[$key];
    }

    $response = [
        'status' => 201,
        'body' => [
            'payment_id' => 'pay_' . (count($idempotencyStore) + 1),
            'amount' => $request['amount'],
        ],
    ];

    $idempotencyStore[$key] = $response;

    return $response;
}

$store = [];
$request = ['idempotency_key' => 'order_123_payment', 'amount' => 1999];

print_r(createPayment($request, $store));
print_r(createPayment($request, $store));

// Output includes the same payment_id both times.

In a real API, the idempotency key should usually be scoped to the authenticated account and endpoint. Two different users should not collide because they chose the same key.

Request fingerprinting

A common safety rule is: the same idempotency key must not be reused with a different request body. Store a fingerprint of the original request and reject mismatches.

<?php

declare(strict_types=1);

function requestFingerprint(array $request): string
{
    ksort($request);

    return hash('sha256', json_encode($request, JSON_THROW_ON_ERROR));
}

$first = ['amount' => 1999, 'currency' => 'GBP'];
$second = ['currency' => 'GBP', 'amount' => 1999];
$third = ['amount' => 2999, 'currency' => 'GBP'];

var_dump(requestFingerprint($first) === requestFingerprint($second));
var_dump(requestFingerprint($first) === requestFingerprint($third));

// Prints:
// bool(true)
// bool(false)

Do not use fingerprints as a substitute for validation. Validate the request first, then store enough information to prove that a repeated key is the same logical operation.

Retry design

Good retry design needs limits:

• Retry temporary failures only.
• Use a maximum attempt count.
• Use backoff, and respect Retry-After where available.
• Do not retry invalid input or authentication failures.
• Require idempotency for operations with side effects.

<?php

declare(strict_types=1);

function retryableByMethod(string $method): bool
{
    return in_array(strtoupper($method), ['GET', 'HEAD', 'PUT', 'DELETE'], true);
}

function canClientRetry(string $method, int $statusCode, bool $hasIdempotencyKey): bool
{
    if (!in_array($statusCode, [408, 429, 502, 503, 504], true)) {
        return false;
    }

    if (retryableByMethod($method)) {
        return true;
    }

    return strtoupper($method) === 'POST' && $hasIdempotencyKey;
}

var_dump(canClientRetry('POST', 503, false));
var_dump(canClientRetry('POST', 503, true));
var_dump(canClientRetry('POST', 422, true));

// Prints:
// bool(false)
// bool(true)
// bool(false)

Server-side race conditions

Idempotency must be enforced atomically. If two identical requests arrive at the same time, both may check "does this key exist?" before either writes it. In real applications, use database constraints, transactions, locks, or a storage system with atomic insert-if-absent behaviour.

An in-memory array is fine for learning the idea, but it is not enough for production PHP running across multiple workers or servers.

The storage choice should match the side effect being protected. A Redis key with SET NX and an expiry can be enough for a low-risk operation where a short retry window is acceptable. A payment, order, booking, or account-provisioning command usually belongs in the same durable database boundary as the business record, with a unique constraint on the scoped idempotency key and an explicit status column. If the idempotency record and the created business record can commit independently, there is still a crash window where the operation succeeds but the replay guard is missing.

Treat the idempotency record as part of the operation, not as logging. It should be written before or during the protected work in a way that prevents another worker from starting the same logical operation. Many systems insert a started record first, commit it, then let the command continue while later updates attach the final response or created resource ID. That design makes concurrent retries see that the key is already in use, even before the first attempt has finished.

Be careful with caches that evict keys under memory pressure. Eviction can turn a duplicate retry into a new operation. If the product promise says "this key is valid for 24 hours", the backing store must be configured so records survive for that window or fail closed when it cannot guarantee retention.

Idempotent Does Not Mean Nothing Changes Elsewhere

HTTP idempotency means repeating the same intended request has the same intended effect on the target resource. It does not mean the response bytes must always be identical. A GET response can change because another user updated the resource. A repeated DELETE may return 204 the first time and 404 later depending on the API convention, while the resource remains deleted.

Design against the business effect. The danger with a timed-out payment request is not that the second HTTP response differs; it is that the customer might be charged twice.

Safe methods and idempotent methods are also different. GET and HEAD are safe: they are intended for retrieval. PUT and DELETE are idempotent but not safe because they intentionally change state.

Scope Keys To The Caller And Operation

An idempotency key should not be globally unique across every customer and endpoint unless the API explicitly designs it that way. A practical scope is authenticated account, route or operation, and key.

<?php

declare(strict_types=1);

function idempotencyScope(string $accountId, string $operation, string $key): string
{
    return hash('sha256', $accountId . '|' . $operation . '|' . $key);
}

echo substr(idempotencyScope('acct_1', 'payments.create', 'checkout_123'), 0, 12) . PHP_EOL;
echo substr(idempotencyScope('acct_2', 'payments.create', 'checkout_123'), 0, 12) . PHP_EOL;

Two accounts may both choose checkout_123; they must not collide. A key used for payments.create should not accidentally replay a response from tickets.create.

Set length and character rules. Extremely large keys can be abused for storage or logging problems. Reject empty keys and normalize header casing through the framework rather than relying on one exact array key spelling.

Store The Request Fingerprint And Response

A robust idempotency record stores more than the key:

• scoped key
• request fingerprint
• operation name
• authenticated actor or account
• status such as started, completed, or failed
• HTTP status and response body for completed operations
• created time and expiry time
• optional resource ID created by the operation
• safe error category for failed attempts

Replaying the same request should return the original completed response. Reusing the same key with a different request fingerprint should return a conflict, commonly 409 Conflict.

Do not fingerprint an unvalidated raw array whose key order or numeric representation changes unpredictably. Parse and validate input first, then fingerprint a canonical representation of the operation.

<?php

declare(strict_types=1);

function canonicalFingerprint(array $validated): string
{
    ksort($validated);

    return hash('sha256', json_encode($validated, JSON_THROW_ON_ERROR));
}

$first = canonicalFingerprint(['currency' => 'GBP', 'amount_minor' => 1999]);
$second = canonicalFingerprint(['amount_minor' => 1999, 'currency' => 'GBP']);

echo $first === $second ? 'same' : 'different';
echo PHP_EOL;

// Prints:
// same

Nested arrays require recursive canonicalization if their order is not meaningful. Lists, however, may be order-sensitive and should not be sorted blindly.

In-Progress Requests Need A Policy

A retry may arrive while the first request is still processing. Returning the final response is impossible until the first attempt completes.

Common policies are:

• return 409 Conflict with an idempotency_key_in_use error
• return 202 Accepted with a status URL
• wait briefly for the first attempt to finish, then replay the stored response

Choose one and document it. Do not run the operation a second time.

<?php

declare(strict_types=1);

function idempotencyStatusResponse(string $recordStatus): array
{
    return match ($recordStatus) {
        'completed' => ['status' => 200, 'code' => 'replay_stored_response'],
        'started' => ['status' => 409, 'code' => 'idempotency_key_in_use'],
        'failed_retryable' => ['status' => 503, 'code' => 'try_again_later'],
        default => ['status' => 500, 'code' => 'idempotency_state_invalid'],
    };
}

foreach (['completed', 'started', 'failed_retryable'] as $status) {
    $response = idempotencyStatusResponse($status);
    echo $status . ': ' . $response['status'] . ' ' . $response['code'] . PHP_EOL;
}

// Prints:
// completed: 200 replay_stored_response
// started: 409 idempotency_key_in_use
// failed_retryable: 503 try_again_later

The stored response for a completed 201 Created operation should usually preserve 201, not become 200, unless the API documents replay responses differently.

Failed Operations Are Subtle

If validation fails before any side effect, the server may not need to store the key. A client can fix the request and send it again with a new or same key according to the API convention.

If the operation reaches an unknown state after partial work, the server should record enough state to prevent duplicate side effects. For example, a payment provider timeout after card authorization may require a status lookup or reconciliation before deciding whether to replay, fail, or return pending status.

Never store an idempotency record after the side effect without connecting both operations transactionally or with a provider guarantee. Otherwise a crash between side effect and key storage can still create duplicates on retry.

TTLs And Cleanup

Idempotency records do not need to live forever, but their lifetime must match the retry window and business risk. Payment and order creation keys often need longer retention than a low-risk notification preference update.

When a key expires, a repeated request may create a new operation. Document the retention window so clients do not assume old keys are permanent. Expiry cleanup should not remove records still needed for reconciliation, disputes, or audit requirements.

Avoid returning a stored response whose referenced resource has been deleted or transformed unless the API contract allows it. Some systems store a response snapshot precisely so replay does not depend on current resource state.

Client Responsibilities

The client should generate a high-entropy or business-unique key before the first attempt and reuse the exact same key for retries of that logical operation. A new key represents a new operation.

For browser or mobile clients, generate the key before submitting the form and persist it until the outcome is known. If the app crashes after sending the request, it can recover the same key and ask for the result rather than creating another order.

Clients should not reuse a key for a different cart, amount, recipient, or form submission. Servers must still protect themselves with fingerprints because clients make mistakes.

Idempotency Across Queues And Workers

An HTTP endpoint may accept a command and enqueue work. The idempotency key should protect both acceptance and worker execution. If the queue message is delivered twice, the worker should use a unique business key, created resource ID, or processed-message table to avoid repeating side effects.

For create operations, a unique constraint on a business operation ID can be more important than an HTTP key alone. Example: one checkout ID creates at most one payment authorization. The HTTP key helps with request replay; the checkout ID protects the business invariant.

What To Check

Before moving on, make sure you can:

• explain why a timeout can lead to duplicate operations
• distinguish safe methods from idempotent methods
• scope idempotency keys by account and operation
• store request fingerprints and completed responses
• reject key reuse with different input
• handle in-progress records without running the operation again
• make idempotency storage durable and atomic
• choose TTLs based on retry window and business risk
• design both client and worker behavior around one logical operation