Investigation 032 - Platform Evolution Principles

The declaration exists.
Nothing makes it real.

INV-031 let a domain describe a ReplicatedDatabase with replicas = 3. The platform preserves that declaration as desired state. Nothing about storing it creates three database replicas. The Controller contract already explains how observed state moves toward desired state. The unresolved question is who should provide that worker when the platform does not own the domain's invariant.

Begin the investigation down
Desired state
replicas = 3
Reality
0 replicas
Behaviour owner
undetermined?

Author's Note

This is not an investigation into Operators.

We already know that controllers exist, that reconciliation exists, and that INV-031 made the resource vocabulary extensible. We inherit the Controller contract from INV-002 and INV-003 rather than rederiving it here. This investigation asks who should own the software that makes a domain-specific resource real, once the owner of the invariant is no longer the owner of the platform itself.

We will put the domain behaviour where the platform already lives, and see why that design is attractive. Then we will give the domain ownership of its behaviour, and ask whether ownership has really separated, or whether we have merely moved the source code while leaving the important boundaries unchanged.

The worker that carries out that behaviour observes desired state, observes a separate external reality, and acts on both. Its observation remains delayed and partial exactly as earlier investigations established, and a platform cannot always tell whether an absent worker is slow or gone. We will not rediscover controllers here.

Who should own the software that makes a domain-specific resource real?

Prologue

Something has to act on the statement.

A team creates ReplicatedDatabase orders-production with replicas = 3. The platform accepts it. Nothing is wrong with the resource, and nothing is missing from the declaration. And yet the database does not exist. There is only a statement saying that three replicas should exist.

Foundation

The platform already knows how to store a domain-specific declaration as authoritative desired state.

Assumption

For a small system with one team, one release process, and rarely changing behaviour, putting that behaviour inside the platform keeps everything close together.

Incident

The platform begins supporting many domains. The database team understands what a healthy database means. The certificate team understands certificate renewal. The platform understands neither.

Mystery

Should the domain also supply the behaviour, and does owning the source code alone make that behaviour independent?

First Principles

A resource is a statement, not an action.

The declaration does not create a database, start a process, allocate storage, or repair an unhealthy instance. Someone must own the knowledge required to make it meaningful, and that ownership question is separate from where the software carrying that knowledge should live.

Resource

A statement of intent

The declaration expresses an intended condition. It does not perform an action.

Meaning

Domain concern

Someone must know what observations matter and which corrective actions are appropriate.

The Controller contract

Already exists

Observe, interpret, calculate, act, repeat. INV-032 asks who should provide that loop, not whether it should exist.

Code ownership

≠ operational ownership

The database team may own the source. The platform may still build, load, run, and upgrade it.

The smallest platform

Keep the behaviour local

One lifecycle, one compatibility boundary, one failure boundary, no additional participant.

Where pressure begins

Domains stop moving together

Different owners, different release cadences, and different operational risk, combined by the naive architecture anyway.

If the domain owns the invariant, should the platform also own the software that implements it?

Naive Architecture

Embed every domain controller in the platform.

The database controller, the certificate controller, and the device controller all live inside the platform. It starts them, provides their state and observation, manages their lifecycle, and releases them together with everything else. There is one operational lifecycle, and no ambiguity about compatibility.

Domain requirementdatabase health, certificate renewal
Platform corecontroller code embedded
Platform releaseone operational lifecycle

The Architecture That Almost Worked

The domain supplies code. The platform still hosts it.

A database team can own the database controller's implementation. The platform no longer needs to understand database semantics merely to execute it, only to host it. The controller is statically bundled and released as part of the host platform, sharing its runtime and process.

What it preserves

One operational lifecycle

No additional runtime, service, or communication boundary. The controller can use the host's existing internal structures directly.

What it assumes

Authorship is ownership

Source-code ownership moved to the domain team. Loading, compatibility, resource sharing, and releaseability are assumed acceptable without yet being tested.

The source boundary moved. We have not yet tested whether the lifecycle, runtime, or failure boundary moved with it.

Breaking Our Design

Four independent pressures test one boundary.

Each experiment starts from its own clean state and tests one prediction about the architecture that almost worked. Read the pressure and prediction, then run the experiment to find out what actually happens.

EPISODE 01

The Release Boundary That Did Not Move

Pressure. The database team owns its controller's implementation, but the platform still bundles and releases it.

Prediction. If the domain fixes its own implementation, it should be able to release that fix independently of the platform.

Host a domain-authored controller, fix its implementation, then attempt an independent release.

Domain controllernot established
Implementation fixnone
Independent releasenot attempted
Outcomenot observed
Prediction: a domain-owned fix should release independently of the platform. Establish the hosted controller to test it.
EPISODE 02

The Shared Failure Boundary

Pressure. The domain owns its controller's responsibility independently, but the controller still shares the platform's process and resources.

Prediction. If the domain's responsibility is independent, a failure inside its controller should stay contained to that responsibility.

Establish independent responsibility in a shared process, trigger a controller failure, then observe unrelated impact.

Shared processnot established
Controller failurenot triggered
Unrelated responsibilitynot observed
Outcomenot observed
Prediction: independent responsibility should stay contained. Establish independent responsibility in the shared process to test it.
EPISODE 03

The Private Interface Trap

Pressure. The controller now runs in its own process, and depends on the platform's private cache, object layout, and queue to keep working.

Prediction. If the controller is physically separate, the platform's internal implementation should be free to change without affecting it.

Move the controller to a separate process, depend on private internals, change the platform's internal implementation, then test compatibility.

Separate processnot established
Private dependencynone
Internal implementationunchanged
Compatibility testnot attempted
Prediction: physical separation alone should make the controller safe from internal platform changes. Move the controller to a separate process to test it.
EPISODE 04

The Independent Worker Under Uncertainty

Pressure. The worker participates through the platform's public resource API. It disappears, and returns with a stale observation while another writer has already acted.

Prediction. If the worker rejoins with its last known observation, it should be able to resume exactly where it left off.

Observe version 10, remove the worker, update authoritative state to version 11, return the stale worker, attempt a stale write, then reread and conclude.

Workernot participating
Authoritative version10 / desired 3
Conditional writenot attempted
Outcomenot observed
Prediction: a returning worker should be able to resume from its last observation. Observe version 10 to test it.
Review the four experiments

    The Turning Point

    Stop asking who owns everything.
    Ask for the smallest participation boundary.

    Domain invariantowned somewhere
    Participation boundary?undetermined split
    Platform guaranteesowned somewhere
    Four failures point at a boundary, not yet its exact split. Chapter 07 formalizes the contract once all four experiments are complete.

    The Domain Automation Participation Contract

    What is the smallest boundary that survives?

    Contract not yet earned. Complete all four experiments before formalizing the contract.

    Only Now: Kubernetes

    Domain controllers realize the participation contract.

    A domain controller operates against resources exposed by the Kubernetes API, using the same resourceVersion-checked updates and bounded watch mechanism available to built-in controllers. When such a controller encodes domain operational knowledge around one or more custom resources, the combination is commonly described as the Operator pattern - a naming convention for the participation boundary derived here, not a new architectural mechanism.

    The custom resource the controller reconciles inherits the extensible resource boundary from INV-031: authoritative persistence, structural conformance, resourceVersion-checked concurrency, and bounded, relist-aware watches.

    This does not cover how a controller is packaged, deployed, retried, backed off, or how it wins leadership among replicas - those are operational choices layered on top of the participation contract. It also does not cover authentication, authorization, or admission - deciding whether a proposed change may become authoritative desired state remains a separate, later question.

    Custom resource

    Authoritative desired state

    The extension boundary INV-031 derived; the domain controller reconciles it, not defines it.

    Domain controller

    Observes and reconciles

    Uses the same resourceVersion-checked updates and watches as built-in controllers.

    Operator pattern

    A naming convention

    Domain operational knowledge packaged around a custom resource, not a new mechanism.

    Engineering Reflection

    Timeless Engineering Principle

    Separate ownership through a stable participation contract around authoritative state, rather than making the domain implementation part of the platform's private machinery.

    Architectural Honesty

    Keep automation embedded

    One team owns the platform and the domain invariant, coordinated releases are intentional, shared process consequences are acceptable, and workloads are predictable.

    Separate domain automation

    Owners and release cadences differ, failure consequences deserve isolation, and another independently operated component costs less than the coupling it removes.

    Costs Accepted

    Another component

    The participant must be deployed, observed, upgraded, diagnosed, and recovered.

    Distributed availability

    The worker can be slow, unreachable, or partitioned without the platform immediately knowing why.

    Convergence liveness

    Desired state remains while domain progress pauses without a capable participant.

    Contract maintenance

    The platform must preserve the public semantics independent participants legitimately depend on.

    Compatibility management

    Host and participant versions still need a defined compatibility relationship.

    Stale observation & concurrent writes

    Participants reason from delayed observation and rely on inherited concurrency guarantees.

    Recovery evidence

    Some domains need durable operation identifiers, idempotency, or checkpoints.

    Explicit boundaries

    Authentication, authorization, admission, packaging, and deployment remain separate concerns.

    Investigation Exercise

    Separate safety from progress.

    Prediction

    Choose a platform hosting domain automation. Predict which property survives if the worker disappears, and which progress stops.

    Experiment

    Give the domain ownership of the implementation, separate the process, replace private dependencies with public participation semantics, then remove every capable worker and return one with no trusted memory.

    Observation

    Record whether authoritative intent remained, whether convergence progressed or paused, and whether the returning worker recalculated from current state rather than memory.

    Reflection

    Identify which coupling was intentional, which guarantees were inherited rather than invented, and which concerns remain outside the participation contract.

    o o o
    Run the synthesis trace after writing your prediction.

    Bridge to INV-033

    A structurally valid proposal
    may still violate independent policy.

    The domain controller can converge accepted state.

    Its resource conforms to the structure INV-031 established.

    Structural legitimacy is not the same as policy acceptance.

    That decision happens before authoritative intent exists.

    Who should decide, and where, without changing the platform's core request-processing loop?

    The platform preserves authoritative desired state while independently owned domain automation reconciles it through a stable participation contract, leaving policy over proposed state unresolved.

    Next Investigation

    INV-033 - The Admission Problem

    How can policy influence what becomes authoritative desired state without modifying the platform's core request-processing loop?

    Intellectual Lineage

    This investigation inherits reconciliation, controller decomposition, durable desired state, and bounded observation from earlier investigations. Kubernetes documents controllers as control loops and the Operator pattern as domain-specific operational knowledge encoded in software. Domain controllers are one realization of the broader participation contract derived here; this investigation does not claim that Kubernetes invented independently owned automation or reconciliation.

    Deliberate Simplifications Ledger

    Controller packaging, deployment, and tuningFuture: Controller Operations (unassigned)
    Authentication and authorization mechanismsFuture: Automation Authorization (unassigned)
    Policy ownership and request-handling semanticsINV-033 - The Admission Problem
    External side-effect atomicity and recovery protocolsFuture: Domain Operation Recovery (unassigned)
    Kubernetes admission terminology and implementationINV-033 - The Admission Problem

    Sources