What we inherit
Level-triggered reconciliation
Correctness comes from reading current state and reducing drift, not from trusting an event history.
INVESTIGATION 004 / THE SILENT CLUSTER
Core Control Plane Principles
How Does a Controller Know When to Look?
A Pod disappears between two observations. The controller owns the invariant and knows how to restore it. Nothing in the architecture tells the controller that the world changed.
Who announces that reality moved?
HAS
ANYTHING
CHANGED?
ANYTHING
CHANGED?
Prologue · The Unanswered Question
The worker can act. It cannot notice.
A replica count changes. A machine disappears. A new object is accepted. Every event creates possible drift, but no controller is automatically aware of it.
Desired Statethree should exist
Controllerready to reconcile
Realitytwo exist now
A controller cannot correct a difference it has not observed.
The obvious solution is request-response. Ask the source of truth for the latest state. Compare. Wait. Ask again.
So let us build the polling machine and allow it to succeed.
First Principles
Observation has an economic shape.
An observer can spend work because time passed or because reality changed. Those two cost models look similar at one controller and radically different at five hundred.
Time-driven work
Ask on a schedule.
Every interval creates a request whether the system changed or remained perfectly quiet.
Change-driven work
React to meaningful movement.
Stable reality stays quiet. Activity rises only when state actually changes.
Polling generates work because time passed. Watching generates work because reality changed.
The Naive Architecture
The Polling Machine
Each controller periodically asks the source of truth for the current objects relevant to its invariant. It is simple, correct, and initially cheap.
Controllerwait for interval
API Serverreturn current state
Reconcileonly if drift exists
The first success
One controller asks every ten seconds. A Pod fails. The next request discovers it. Reconciliation restores the invariant.
The architecture is correct. The delay is bounded by the interval, and the next request naturally recovers from a failed request.
Poll requests0
Useful answers (upper bound)0
Proofs of nothing0
Worst latency10s
Bounded model: every listed change is relevant to every controller and is discovered by a distinct next poll. Real relevance and change coalescing can only reduce the useful-answer count.
Ready: Configure the observers, then run sixty seconds of quiet and change.
The Architecture That Almost Worked
Make the interval shorter.
Slow discovery is uncomfortable, so the natural improvement is to ask more often. Latency falls. Request volume rises even when reality is still.
Ten-second interval
Bounded delay
One controller makes six requests each minute. A quiet answer still costs one complete exchange.
One-second interval
Faster, not freer
The same controller now makes sixty requests each minute. Ninety-nine quiet controllers multiply the proof-of-nothing work.
Ready: Place a change between two poll boundaries.
Scale Changes the Economics
The quiet cluster stays busy.
With 500 controllers polling every ten seconds, the source answers 3,000 requests each minute before one meaningful change occurs. The architecture spends most of its effort proving stability.
Controllers500
Poll interval10s
Requests / minute3,000
Changes / minute1
The system is computationally active while operationally idle.
The numbers are illustrative arithmetic, not measurements or a prescribed interval. They expose the order-of-magnitude pressure: observers multiplied by elapsed intervals, regardless of meaningful activity.
Breaking Our Design
Five failures of asking.
Polling remains correct. Each failure reveals why its observation cost stops matching the work the system actually needs.
FAILURE 01
The Question That Never Changed
Every interval repeats the same request against unchanged state.
Time, not change, creates work.FAILURE 02
The Controllers Multiply
Independent responsibility creates more observers, each with its own schedule.
Observation load grows with observer count.FAILURE 03
The Empty Answer Is Expensive
A response saying “nothing changed” still consumes transport, authorization, serialization, and processing.
No-op answers are not free.FAILURE 04
Freshness Demands More Questions
Shorter intervals improve discovery latency by increasing the request rate everywhere.
Polling trades cost directly for freshness.FAILURE 05
The System Wasted Work
Most observation work establishes that stable reality remained stable.
Successful growth disconnects effort from meaningful activity.The Turning Point
Stop searching for change.
Let the source of truth announce meaningful change.
Source of Truthstate changes
Continuous Streamannounce the change
Controllerre-read and reconcile
Responsibility moves to the component best positioned to know when accepted state changes. Observers remain idle while reality is stable.
Communication becomes proportional to meaningful activity rather than elapsed time.
The Architectural Reveal
The architecture earns its nameKubernetes calls it a
Watch.
Watch.
The world should speak when it changes. Observers should listen.
The Honest Failure
A stream can break.
A Watch accelerates observation. It does not guarantee that every announcement arrives exactly once forever. Correctness must remain level-triggered.
Source of Truth
replicas = 3
replicas = 3
Controller View
replicas = 3
replicas = 3
ConnectionClosed
Events delivered0
Events missed0
Views matchYes
Ready: Open the stream, change state, then interrupt observation.
The Observation Fence
A Watch says “something changed; look again.” It does not transfer correctness into the event stream. Connections drop, processes restart, and observers can miss announcements.
Level-triggered reconciliation survives because it trusts current state. A missed hint delays correction; a full re-read can restore knowledge.
The Watch Contract
The source of truth announces meaningful change. Observers react instead of rediscovering it on a timer.
Announce change.
The source publishes accepted changes because it is best positioned to know they occurred.
Maintain interest.
An observer keeps an ongoing relationship rather than creating a fresh question each interval.
Use events as hints.
An announcement schedules useful work; it does not become the source of correctness.
Trust current state.
Reconciliation reads the present report, so a missed event delays correctness rather than destroying it.
Engineering Reflection
Align effort with meaning.
A scalable architecture should spend resources reacting to meaningful change, not repeatedly proving that nothing changed.
Architectural honesty
Keep polling
For a handful of observers and resources, polling is simpler to operate, inspect, and recover than a long-lived stream.
When watches earn their cost
Listen continuously
When observers multiplied by resources divided by interval begins to rival useful work, change-driven observation earns its complexity.
Costs Accepted
ConnectionsLong-lived relationships must be maintained.
InterruptionA broken stream must be detected.
ResumptionAn observer needs a safe way to regain current knowledge.
ComplexityEvent delivery joins, but never replaces, reconciliation.
Investigation Exercise
- Prediction
Decide whether a watch reports on a timer or when state changes. - Experiment
Record a prediction, then run polling and watch against the same three changes. - Observation
Compare requests, notifications, and discovery delay. - Reflection
Explain why notification accelerates a level-triggered loop rather than replacing it.
PredictionNone
Polling requests0
Watch notices0
Fastest discoveryHidden
Prediction: Choose one model before running the comparison.
The Next Mystery
Every controller is listening.
Should each controller maintain its own connection?
How many times should one Pod change cross the network?
How many identical copies of state should one process hold?
Can observation itself become shared infrastructure?
Deliberate Simplifications Ledger
- How a watcher resumes after losing its placeINV-009 (Versioned Truth)
- How many controllers avoid duplicating identical observationINV-005 (Observe Once, Share Many)
- Why the watched stream reflects one agreed truthINV-007 / INV-008