Define the expected work, not only the timer
Write the contract before opening a deployment screen:
| Field | Question to answer |
|---|---|
| Slot | Which scheduled interval or queue event is expected, and in which timezone? |
| Operation | What business operation is being attempted? |
| Identity | Which idempotency or operation key lets an operator find the result? |
| Concurrency | May two runs for the same operation overlap? |
| Retry | Which failures are retryable, and how many attempts are allowed? |
| Result | What authoritative record says the operation succeeded or failed? |
| Owner | Who investigates and who is the backup? |
“The job runs every five minutes” is not a sufficient result contract. A
reviewer should be able to ask for slot 10:05, find its attempts, and see
whether the application recorded a terminal business result or only a handler
invocation.
Exercise the failure matrix
Use safe synthetic data in an approved disposable or non-production environment. The test card in the evidence packet uses these cases:
| Case | Deliberate setup | Evidence that should remain |
|---|---|---|
| Normal run | Allow one expected slot to complete | One terminal result with slot and operation identity |
| Missed slot | Suppress or disable one invocation | A visible missing slot with an owner and next action |
| Duplicate delivery | Deliver one slot twice | Both attempts, their identities, and the rule that prevents unsafe replay |
| Overlap | Hold one run open while the next eligible run starts | Overlap evidence and an explicit concurrency decision |
| Retry | Fail the first attempt | Attempt count, retry reason, and whether the operation key stayed safe |
| Unknown outcome | Hide the terminal response | A reconciliation task, not an automatic retry |
The local fixture checks these states from supplied run records. Run it with:
npm test --prefix sites/howtox.com/evidence/P90
A passing local test means the record model can expose the cases. It does not mean your scheduler delivered a request or that your database side effect is idempotent.
Make retries and overlaps observable
Retries are not merely a number in a log. Record the expected slot, attempt number, start and finish time, operation key, outcome, and the reason for a retry. If two attempts can be live at once, record both intervals and decide whether the second should wait, exit, or continue under a safe operation key.
Do not hide duplicate delivery by overwriting the first row. A duplicate is useful evidence: it may reveal a provider retry, a timeout after a completed side effect, a manual replay, or an application race. The right response depends on the operation. A read can often be repeated; sending a notification or charging a customer needs a different contract.
Reconcile before declaring success or retrying
An unknown result means that the caller lacks an authoritative terminal record. It is not proof of failure. First look up the operation key in the durable result source, then record the reconciliation result and who made the decision. Only after that check should an operator choose a retry, containment, or no-op.
This separation also keeps code rollback and data recovery distinct. Restoring an old handler does not undo a side effect that already happened. A runbook should say who can disable new work, who can reconcile the existing operation, and where the evidence is retained.
Launch gate and stop condition
Before launch, have the product owner review the completed card and confirm:
- the environment and synthetic data are approved;
- normal, missed, duplicate, overlap, retry, failure, and unknown cases have an observable record;
- the operator and backup can find the evidence;
- the retry and reconciliation rules are explicit; and
- containment does not require editing production data by guesswork.
Stop if the team can only say “the cron fired,” cannot distinguish a missed slot from a failed business operation, or has no safe answer for an unknown outcome. The draft's local fixture is inspectable evidence for the test contract, not a completed application exercise.