What we inherit
Reconciliation is no longer mysterious.
Desired state, reported reality, corrective action, and repetition already form the contract.
INVESTIGATION 003 / THE HIDDEN WORKER
Core Control Plane Principles
A Contract Cannot Do the Work.
A workload disappears. Another appears. The reconciliation promise explains what must remain true, but a promise cannot observe, decide, or act.
Who stays awake after the command has ended?
ONE
BRAIN
ObserveCompareDecideCorrectBRAIN
Prologue · The Hidden Worker
A factory does not operate itself.
Conveyor belts move, robotic arms assemble, and finished goods leave the warehouse. The machinery looks automatic only because the workers behind its decisions are easy to overlook.
The illusion
The system healed itself.
A workload disappeared. Another appeared. A desired count changed. More work arrived. Convenient language hides the actor.
Read between the lines
Every verb requires someone.
Observation, comparison, creation, and deletion are actions. A contract specifies the destination; a participant must continuously perform the journey.
A contract defines what must happen. It never lays a brick.
Some participant must remain on duty while administrators sleep, after commands end, and after individual processes restart. The simplest answer is one central worker responsible for everything.
So let us build that central brain and allow reality to test it.
First Principles
Responsibility is the boundary.
Large systems rarely survive by making one participant understand everything. They survive by making every responsibility small enough to name, reason about, and change independently.
The tempting answer
Give one process complete visibility.
One log, one deployment, one place for every decision. For a small system, that economy is real.
The unresolved question
What does that process own?
Ownership means the invariant a component promises to protect, not merely the actions it happens to perform.
Independent problems deserve independent solutions because they change for independent reasons.
The Naive Architecture
The Central Brain
Build one long-running process. Let every change flow in, every decision flow out, and every correction begin in the same place.
Desired Statewhat should exist
Central Brainunderstands every rule
Corrective Actionchange reality
Noticed
A missing workload is observed and replaced.
Adjusted
A new desired count produces the missing work.
Recovered
Drift is corrected without a human reissuing a command.
The first success
It works. There is one executable to deploy, one set of logs to inspect, and one obvious home for each new feature. Early success proves that centralization is a sensible beginning.
It does not prove that one responsibility boundary can absorb an unknown future.
Central Brain
4 responsibilities
4 responsibilities
Responsibilities04
Ownership boundaries01
Changed modules00
Modules in release04
The Architecture That Almost Worked
Every capability has an obvious home.
One new rule. One additional module. One more special case. The cluster remains healthy while the design quietly accumulates knowledge.
Nothing fails on a dashboard. The cost appears in broader reviews, larger regression suites, synchronized releases, and engineers learning unrelated domains before making local changes.
The system is not becoming computationally overloaded. It is becoming over-responsible.
The platform grows outward. The architecture grows inward.
Breaking Our Design
Seven cases. One ownership flaw.
The software continues working. Each incident breaks a different assumption about whether one decision-maker can remain the home of every future invariant.
CASE 01
Every New Resource Needs More Code
- Incident
- A resource arrives with its own lifecycle, failure rules, and definition of correctness. The same process must learn all of it.
- Broken assumption
- Capabilities can grow forever inside one bounded responsibility.
- Discovery
- Capabilities should grow. Individual responsibilities should not.
CASE 02
The Feature That Broke Another Feature
- Incident
- A local bug fix changes shared decision logic. An unrelated resource begins recovering differently.
- Broken assumption
- Independent business rules remain independent inside a shared decision-maker.
- Discovery
- Shared ownership creates coupling even when nobody intended interaction.
CASE 03
The Team That Could Not Work Independently
- Incident
- Recovery, storage, and rollout teams all change the same component. Parallel work becomes coordination.
- Broken assumption
- More engineers increase capacity around one shared responsibility boundary.
- Discovery
- The bottleneck is architectural contention, not computing power.
CASE 04
The Resource Nobody Planned For
- Incident
- An unforeseen resource forces the core to learn another world. Extension still means changing the center.
- Broken assumption
- Flexible code alone can make an ownership model extensible.
- Discovery
- New responsibilities must arrive without modifying unrelated owners.
CASE 05
The Release That Became Risky
- Incident
- A small rule change ships inside the executable that protects every invariant. The local edit inherits a global blast radius.
- Broken assumption
- A local code change implies a local operational risk.
- Discovery
- Blast radius follows the ownership and deployment boundary.
CASE 06
Breaking the Central Brain
- Incident
- Refactoring, testing, and faster hardware cannot remove the burden while every invariant still belongs to one place.
- Broken assumption
- The problem is implementation quality.
- Discovery
- The platform must be divided by responsibility, not implementation.
CASE 07
Inventing Independent Workers
- Incident
- The final attempt refuses to enlarge the brain. One small worker receives one invariant, and the next responsibility receives another worker.
- Broken assumption
- One coherent platform requires one participant to understand the whole.
- Discovery
- Global behavior can emerge from many narrow, independently responsible workers.
BoundaryCentral
Owned invariants07
Invariants exposed00
DefectNone
The Turning Point
Stop growing the brain.
Add another worker.
Replica count
Protect only the requested population.
Rollout progress
Protect only movement toward a requested version.
Completed work
Protect only the completion invariant.
Machine availability
Protect only the relevant availability decision.
Growth changes direction. Capabilities expand by adding participants; no existing participant must absorb the new rules.
That does not make the architecture fully decentralized. Decision-making is distributed, while the source of shared truth remains central and unresolved. Simultaneous writes and duplicate active copies also remain unresolved.
Workers03
Assigned03
Drifted invariants00
Last worker runNone
The architecture earns its nameKubernetes calls them
Controllers.
Controllers.
ReplicaSet Controller
Decides whether the desired number of replicas exists.
Deployment Controller
Decides whether a rollout should continue.
Job Controller
Decides whether work has completed successfully.
StatefulSet Controller
Protects the invariant for ordered, stateful workloads.
A Controller is defined by the invariant it protects, not merely by the resource it observes.
The Controller Contract
A Controller preserves one invariant through bounded, independent, repeated action.
One invariant.
State one condition of correctness the worker promises to preserve.
Read relevant state.
Use the best available report without claiming perfect present knowledge.
Find the difference.
Compare desired state with the report relevant to this invariant.
Reduce drift.
Compute the corrective action necessary to reduce the difference.
Act only within the owned responsibility.
Do not absorb diagnosis, policy, or decisions that belong to another invariant.
Evolve alone.
Unrelated Controllers should not need to change together.
Remain responsible.
Return to observation for as long as the invariant exists.
The Observation Fence
"Observe reality" is useful shorthand, not a claim of global present knowledge. A Controller reads reports assembled from messages that took time to arrive. Observation is delayed and partial.
This companion does not explain efficient observation, authoritative shared storage, simultaneous-write arbitration, or how one active copy is chosen. Those are later mysteries, not hidden implementation details.
Engineering Reflection
Responsibility is the real scaling unit.
Large systems are built from many small owners with defensible boundaries. Complexity is not eliminated. It is distributed so that each part can remain understandable.
Architectural honesty
Keep the central process
When one team protects one or two invariants, one process has fewer moving parts to deploy, monitor, and understand.
When decomposition earns its cost
Divide responsibility
When many invariants must evolve and fail independently across teams, narrow owners reduce coupling and blast radius.
Costs Accepted
Many small processes are individually easier to understand but collectively harder to operate than one big one. The architecture accepts more deployments, more monitoring, and more places where failure can originate in exchange for responsibilities that change and fail independently.
DeploymentMore independently shipped components.
ObservationMore participants need reports of shared state.
ContentionIndependent decisions can reach for the same object.
OperationsMore health, ownership, and failure surfaces to understand.
Investigation Exercise
The commands are not the lesson. Predict how one responsibility boundary answers two changes, then identify who converges the resulting Pods.
- Prediction
Decide whether an image change and a replica-count change require the same decision. - Experiment
Create a Deployment, change its image, then change only its replica count. - Observation
The Deployment controller initially reconciles both changes. A template image change creates a new ReplicaSet; a count change scales the active ReplicaSet. The ReplicaSet controller then converges Pods in both cases. - Reflection
One controller can make different decisions within one invariant boundary, while another controller converges the Pods it owns.
Image decisionUnrevealed
Count decisionUnrevealed
Initial reconcilerUnrevealed
Pod convergenceUnrevealed
The Next Mystery
Every Controller must observe.
Does each one repeatedly ask whether anything changed?
What happens when ten observers become hundreds?
How much work is spent receiving the answer "no"?
Could change announce itself instead?
Dividing responsibility creates more observers. In one observation round, 10 independently polling controllers make 10 requests; 500 make 500 requests. This comparison says nothing about how often a round occurs or what policy schedules it. It only exposes the order-of-magnitude pressure on the shared control plane.
Desired state
The invariant that should remain true.
Reported reality
The worker reads the state relevant to its responsibility.
Difference
The worker decides whether its invariant is satisfied.
Bounded action
The worker reduces drift, then returns to observation.
Deliberate Simplifications Ledger
- How independent controllers avoid clobbering the same object on simultaneous writesINV-009 (Versioned Truth)
- What prevents two copies of the same controller from both actingINV-010 (The Single Leader Problem)
- The shared, consistent store every controller reads from and writes toINV-007 (The Source of Truth)
- How a controller observes change efficiently, and without duplicating that observationINV-004 (Watches) and INV-005 (Informers)