One machine stopped responding. Its work vanished. Minutes later, the missing work existed elsewhere, although nobody issued another command.
No engineer restarted it. No application requested it. Who noticed reality had changed?
No new human command appears between disappearance and repair.
We naturally speak in actions: create, delete, restart, scale. Software inherited that language because it works beautifully when a completed action stays complete.
A local program reads input, writes output, and exits. If nothing changes afterward, the command permanently solved the problem.
Processes crash. Machines disappear. Messages are delayed. Operators change their minds. The result begins aging the instant it is produced.
A command describes what should happen once. A distributed system must preserve what should remain true.
That is the quieter mystery left by INV-001. A missing process returned without a second instruction. Before asking how, we have to build the architecture most engineers would build first.
A client sends an instruction. A server forwards it. A machine performs it. Every effect has a visible cause, and every request has an end.
The demonstration works. Three workers appear, the response says success, and the room accepts the architecture.
Then one worker disappears after the request has finished. Nothing happens. The machine knows how to execute an action, but nobody owns preservation of its outcome.
Delivery status: Ready
Control-plane view: All machines responding
Restart status: History contains one command
The control plane sees only silence. From that observation alone, it cannot know why the machine stopped responding.
Each failure looks different. Each asks the command machine to prove a present fact from an incomplete record of the past.
A timeout does not say the request failed. It says the client stopped hearing. Retrying is correct if the request vanished and dangerous if only the reply vanished.
A command proves an attempt, not an outcome.
The first request succeeds, its reply disappears, and a reasonable retry performs the action again. Different participants hold different, internally consistent histories.
Delivery count cannot establish current correctness.
A crash, a partition, overload, and a delayed heartbeat all present the same observation: one machine stopped responding. Silence says communication ended. It does not explain why.
You cannot tell slow from dead from silence alone.
The command processor restarts with empty memory while the workers it created remain. Replaying remembered actions can omit required work or duplicate work already done.
Process memory is not an architectural source of truth.
The ledger can honestly record three successful creations while only two workers exist now. The record is not fraudulent. It answers a question the requirement did not ask.
History may explain reality. It cannot authorize claims about the present.
Stop asking what happened.
The requirement concerns what should exist now. The architecture keeps looking backward.
| Failure | What the record says | What it proves now |
|---|---|---|
| Lost response | An attempt may have executed | Nothing |
| Duplicate delivery | The action ran more than once | Nothing |
| Silent machine | The last report was healthy | Nothing |
| Processor restart | The volatile ledger is empty | Nothing |
Replace an expiring instruction with a persistent statement of intent. Not "create three." Instead: "three should exist."
A command expires when it finishes. The declaration remains true five minutes later, after a process restart, and after a machine disappears.
Failures stop being special cases. Missing work, excess work, a changed image, and a new replica count all become one condition: reality differs from intent.
The intent itself may never be lost. This investigation assumes that shared declaration survives and can be read consistently. How failing machines maintain that shared truth is a different, much harder mystery, deliberately left unopened here.
A declaration alone changes nothing. Something must read a report, compare it with intent, reduce the difference, and return to the beginning.
The difference is the work. The cause can remain unknown.
Observe. Compare. Correct. Repeat. The loop converges; it never claims the world will remain correct forever.
Status: Reality matches intent
Last difference: 0
Last action: No pass yet
Pass count: 0
The observation is never the territory. A report can be delayed, partial, and occasionally wrong. How it becomes trustworthy enough to act on remains a later mystery.
The 1,000-pass demonstration is bounded to an already-satisfied invariant, one reconciler, no concurrent actor, and a fresh, complete report.
A reconciler continuously makes actual state converge toward desired state.
Discover the best available account of current reality, without pretending it is a globally current view.
Compare the report with durable intent. The difference, not an old event, determines whether work remains.
Create missing work, remove excess work, or update stale work. Correct only what the available evidence permits.
Return to observation. Equality is a stable moment, not permanent completion.
"Observe reality" is useful shorthand, but it is not literally possible across many machines. What the loop reads is a report assembled from messages that took time to arrive. It may be late, incomplete, or wrong.
In this bounded lab, an incomplete report postpones correction. That is a teaching constraint, not a claim that all stale actions are harmless. Trust, action preconditions, and arbitration belong to later investigations.
When reality can drift after an instruction completes, preserve the intended invariant and continuously compare it with the best available report of reality.
A one-shot migration, local build, or synchronous calculation has no continuing outcome to preserve. Command and completion remain simpler and better there.
Use continuous convergence when reality changes independently, intent must outlive attempts, failure outcomes are ambiguous, and correctness must be restored without a human re-trigger.
The stronger contract is not free. It exchanges dependence on perfect history for continuous work and additional operating machinery.
The exercise assumes an already-satisfied invariant, one reconciler, no concurrent actor, and a fresh, complete report.
The vocabulary is modern. The architectural idea is not.
A feedback loop observes a process variable, compares it with a setpoint, applies correction, and repeats. A thermostat and this cluster share that closed-loop shape.
Google's large-scale cluster manager let users declare what should run and continuously drove a changing fleet toward those declarations.
Google's later research explored multiple independent schedulers sharing cluster state, exposing the coordination pressure that appears when many actors act concurrently.
Tools change. The loop remains: intent, report, difference, correction.
Who wakes up?
Who observes the report?
Who owns the unfinished correction?
Who keeps doing it after a restart?
The reconciliation contract defines what must remain true. It deliberately says nothing about the component responsible for making it true.
