Investigation 026 - Persistent State Principles

Place the compute.
Can it reach the data?

A healthy node and a healthy volume can still form an impossible placement when each belongs to a different topology.

Begin the investigation down
Compute feasible
Node ANode B
Storage reachable
Node ANode B no
Intersection
Node A

Author's Note

Persistence solved lifetime. It did not solve location.

INV-025 gave storage a platform-owned lifecycle independent of disposable execution. We inherit that contract rather than rediscovering it.

This investigation asks when computation and storage may be placed independently. It stops before provider interfaces, stable workload identity, storage failure detection, and access coordination.

Computation can move. Topology-constrained storage cannot.

Prologue

Every component is healthy. The workload still cannot start.

The scheduler chooses a node with available CPU and memory. The persistent volume remains healthy in another topology. Attachment never becomes possible.

Foundation

Storage outlives execution.

Assumption

Storage can follow any valid compute placement.

Incident

The chosen node cannot reach the required volume.

Mystery

How should a platform place computation when storage has location?

First Principles

Correct execution exists only in the intersection.

Compute feasibility and storage reachability are different constraint sets. A destination is valid only when it belongs to both.

Compute set

Can execution fit?

CPU, memory, and inherited workload placement constraints filter nodes.

Storage set

Can data be reached?

Hardware, zone, or another topology boundary may filter the same nodes.

Correctness

Intersect first

An unreachable node is infeasible, not merely less desirable.

Storage topology is a correctness filter when it makes attachment impossible. Ranking among feasible outcomes remains policy.

Topology observations are reports from a distributed system. They may be delayed or incomplete; no instant global view is assumed.

Naive Architecture

Place computation. Let storage follow.

The scheduler owns workload placement. The storage subsystem owns persistence. Each makes its decision independently, preserving clean responsibility boundaries.

Create Volumestorage lifecycle
Choose Nodecompute constraints
Attach Laterreachability assumed

The Architecture That Almost Worked

Independent owners make independent placements.

For universally reachable storage, this decomposition works. For storage with geography, each owner sees a valid local decision while neither can guarantee a valid combined outcome.

Scheduler

Owns workload placement

It filters compute candidates from the evidence available to it.

Storage subsystem

Owns storage lifecycle

It creates, preserves, and exposes storage within its own constraints.

Independent ownership remains correct. Independent placement does not always remain correct.

Breaking Our Design

Three independent pressures earn one placement contract.

Each expanded experiment starts from its own clean model and stops at its assigned discovery.

EPISODE 01

Storage That Cannot Move

Pressure. Node A in zone-a and Node B in zone-b are compute-feasible. A required healthy volume already exists in zone-a.

Prediction. A scheduler using compute constraints alone may choose Node B and expect attachment to follow.

Experiment

Choose compute, reveal topology, attempt attachment.

Observation

Node B and the volume are healthy.

Failure

Different zones make the pair incompatible.

Discovery

Reachability must be visible to scheduling.

Next pressure

What if topology is visible from the start?

Boundary

No conclusion about commitment timing yet.

Compute-only scheduling against an existing volume.

Node Azone-a, compute-feasible
Node Bzone-b, compute-feasible
Volumehealthy, topology not observed
Scheduler choicepending
Attachmentpending
Prediction: compute-only scheduling may choose either healthy node.

This model assumes the observed zone-a location for illustration. In reality, topology evidence may be delayed or incomplete.

EPISODE 02

Binding Before Scheduling Traps the Scheduler

Pressure. Storage topology is visible from the start. Zone-a compute is full; zone-b compute is available.

Prediction. Binding storage in zone-a before evaluating compute may leave no compatible outcome.

Experiment

Commit storage, then evaluate both constraints.

Observation

Storage reaches zone-a; capacity exists in zone-b.

Failure

The feasible intersection is empty.

Discovery

Compatibility must remain unresolved before commitment.

Next pressure

How should applications express storage need?

Boundary

No application abstraction introduced yet.

Visible evidence arriving before an irreversible choice.

zone-a computefull
zone-b computeavailable
Storageuncommitted, topology visible
Intersectionnot evaluated
Evidencepending
Topology is visible. Test whether visibility alone prevents an impossible commitment.
EPISODE 03

Not All Storage Is Equivalent

Pressure. The application directly names fast-ssd-zone-a. Another environment provides equivalent properties under different provider and tier identities.

Prediction. The application description will fail to migrate even when its actual requirements can be met.

Experiment

Migrate identity, express properties, map locally, coordinate.

Observation

Equivalent capability exists under another name.

Failure

Infrastructure identity couples the application.

Discovery

Applications state properties; platforms map them.

Contract

Coordinate both feasible sets before commitment.

Boundary

Provider participation belongs to INV-027.

Portable intent, local infrastructure, coordinated commitment.

Environment onefast-ssd-zone-a exists
Environment twoequivalent properties, different names
Application requestfast-ssd-zone-a
Platform mappingpending
Commitmentpending
The current environment satisfies the named identity. Attempt migration.
Review the three experiments
  1. Topology must be represented where correctness filtering occurs.
  2. Visibility cannot undo a permanent commitment made before compatibility is known.
  3. Portable requirements let the platform map local infrastructure and coordinate both decisions.

The Turning Point

Evaluate both feasible sets.
Commit neither decision alone.

Compute Constraintsscheduler owned
Feasible Intersectionplatform coordinated
Storage Constraintsstorage owned
Coordination changes how decisions interact. It does not transfer ownership.

The Storage Placement Contract

Five responsibilities. One coordinated outcome.

Contract not yet earned. Complete Episode 3 through coordinated commitment.

Only Now: Kubernetes

StorageClass and delayed binding are a bounded realization.

A StorageClass lets an application request a class of storage properties without directly naming one backend identity. The platform maps that request to infrastructure available in the cluster.

volumeBindingMode: WaitForFirstConsumer delays binding and dynamic provisioning until a consuming Pod exists. That delay allows Pod scheduling constraints to inform storage topology rather than committing storage location independently.

Request

Storage properties

The workload requests a class rather than selecting a provider mechanism.

Scheduling

Volume binding participates

The scheduler considers volume requirements with workload feasibility.

Topology

PV node affinity

At a high level, volume topology constrains which nodes may use a bound volume.

Not every StorageClass or backend is topology constrained. Delayed binding does not guarantee provisioning or attachment will succeed under partial failure, stale evidence, exhausted capacity, or backend failure.

How providers participate belongs to INV-027. Stateful workload identity belongs to INV-028.

Engineering Reflection

Timeless Engineering Principle

Independent ownership does not imply independent placement.

Architectural Honesty

Keep placement independent

When storage is universally reachable or data is ephemeral or reconstructable, coordination may add cost without improving correctness.

Coordinate placement

When one resource constrains where another may exist, both constraints must be respected before commitment.

Costs Accepted

More constraints

The platform evaluates compute and storage feasibility together.

Cross-subsystem evidence

Decisions depend on delayed information from independent owners.

Longer decision paths

Commitment waits while compatibility is established.

Harder diagnosis

Healthy components may still have an empty feasible intersection.

Investigation Exercise

Trace three strategies across two zones.

Prediction

A volume exists in zone-a while free compute exists only in zone-b. Predict each outcome.

Experiment

Compare compute first, storage first, and coordinated placement.

Observation

Ask whether each strategy can eliminate every valid outcome.

Reflection

Separate individual component health from collective feasibility.

o o o
Run the placement trace after writing your prediction.

Bridge to INV-027

Placement is coordinated.
But implementations keep multiplying.

Computation and storage now participate in one placement outcome.

Applications express required properties rather than infrastructure identities.

The scheduler and storage subsystem retain independent ownership.

Local disks, cloud volumes, and storage appliances expose different operations.

The platform cannot grow by learning each implementation directly.

Compute and storage constraints converge before commitment, then multiple storage implementations open the interface question

Next Investigation

INV-027 - The Storage Interface Problem

How can a platform support storage systems it has never seen before without changing its own architecture?

Intellectual Lineage

This investigation inherits constraint filtering, desired-state reconciliation, and independent responsibility from earlier investigations. Its placement lineage includes Borg and Omega, while it applies those inherited ideas to two resources whose placement constraints participate in one outcome. It does not claim that Borg or Omega introduced Kubernetes volume binding or topology-aware provisioning.

Deliberate Simplifications Ledger

Dynamic storage creation after coordinated placementINV-027 - The Storage Interface Problem
Common contracts and communication with external storage implementationsINV-027 - The Storage Interface Problem
How stateful workloads build on these placement guaranteesINV-028 - The Stable Identity Problem
Whether an unresponsive storage system is failed or merely slowStorage Failure Detection (unassigned)
Preventing simultaneous incompatible claims on the same volumeStorage Access Coordination (unassigned)

Sources