Investigation 025 - Persistent State Principles

Replace the process.
Keep the data?

The platform is correct to treat execution as disposable. A database is valuable precisely because its information is not.

Begin the investigation down
Execution
Replacement
Information

Author's Note

Every workload discussed so far could forget.

We inherit disposable Pods, automatic replacement, placement freedom, and governed communication. We do not rediscover those contracts.

This investigation distinguishes an execution-owned writable filesystem from every other filesystem or volume a Pod might mount, and stops before placement, interfaces, and stable identity.

Computation is recreated. Information may need to endure.

Prologue

The resilience mechanism becomes the failure.

A stateless server disappears and a replacement continues. A database returns without the history that made it useful.

Foundation

The platform replaces missing execution.

Assumption

Replacement has no lasting consequence.

Pressure

The workload accumulates irreplaceable information.

Mystery

How can data outlive disposable execution?

First Principles

Lifetime follows ownership, not storage medium.

Moving bytes from memory into files changes durability only when those files belong to something with a longer lifecycle.

Execution

Temporary

A Pod may be created, destroyed, and replaced.

Information

Business lifetime

Required lifetime comes from meaning, not the current process.

Boundary

Independent ownership

Different lifetimes require independently governed resources.

Naive Architecture

Fresh execution is always enough.

For stateless work, starting another instance restores correctness because no valuable local history must survive.

Web Servertemporary
Replacementstarts fresh
Correct Servicenothing lasting lost

The Architecture That Almost Worked

Write database records to files.

Records are visible in the container's writable layer. The medium changed; the owner did not.

Databasewrites records
Container Writable Layerexecution owned
Visible Fileswhile this container exists

Breaking Our Design

Three independent pressures earn one storage lifecycle.

Each experiment begins clean and stops before the next episode's answer.

EPISODE 01

The Filesystem Dies With the Pod

Pressure. A database stores records in its execution-owned writable layer.

Prediction. Files should survive because they are not only in memory.

Experiment

Write, inspect, replace, inspect.

Observation

The replacement layer is empty.

Failure

Files inherited execution ownership.

Discovery

Medium did not change lifetime.

Next pressure

Find a longer-lived owner.

Boundary

No node or volume solution yet.

Database records inside one container's writable layer.

Podpod-a
Container writable layercontainer-a execution owned
Written0
Visible0
Evidencepending

EPISODE 02

Rescheduling Moves Compute, Not Data

Pressure. Records live on Node A's local disk.

Prediction. Surviving process replacement should mean surviving rescheduling.

Experiment

Replace locally, suspect A, schedule on B.

Observation

Local recovery works; B lacks access.

Failure

Compute and data location diverge.

Discovery

Node-local state inherits node constraints.

Next pressure

Own a longer lifecycle.

Boundary

Topology waits for INV-026.

Process replacement is not rescheduling.

Node Aavailable
Node A disk0 records
PodNode A
Local recoverypending
Node B accesspending

EPISODE 03

Applications Cannot Own Storage Lifecycle

Pressure. Four applications each provision, attach, and reclaim storage.

Prediction. Independent ownership should preserve a consistent platform environment.

Experiment

Assign 3 duties to 4 apps.

Observation

12 assignments diverge.

Failure

Collision, orphan, readiness gap.

Discovery

Platform owns lifecycle.

Contract

Resource precedes and outlives execution.

Boundary

No universal attachment claim.

Duplicated ownership fails before lifecycle separation succeeds.

Apps4
Assignments0
Failurepending
Logical resourceabsent
Executionnot started

The Turning Point

Keep execution disposable.
Give storage its own lifecycle.

Application Needexpressed
Platform Ownershipprovision, attach, reclaim
Logical Storageindependent lifecycle

The Persistent Storage Contract

Five responsibilities. One platform owner.

Contract not yet earned. Complete Episode 3 through replacement request.

Execution remains replaceable because persistent storage has an independently owned lifecycle.

Contract 1

Independent existence

Execution start and stop do not implicitly create or destroy logical storage.

Contract 2

Process replacement

Replacing computation must not become data destruction.

Contract 3

Same logical resource

Across rescheduling, storage remains the same logical resource. This is an architectural declaration, not proof: topology, partitions, partial failures, and concurrent writes constrain implementation and remain deferred.

Contract 4

Application expresses need

The platform owns provisioning, attachment, and reclamation.

Contract 5

Decoupled lifetimes

Execution may change without redefining persistent information's lifecycle.

Responsibility is assigned without promising physical durability under every failure.

Only Now: Kubernetes

PersistentVolume and PersistentVolumeClaim are a bounded realization.

A PersistentVolume represents cluster storage with a lifecycle independent of a Pod. A PersistentVolumeClaim is a workload's request for storage. The application expresses need; the platform owns lifecycle work.

This differs from the ordinary ephemeral container writable filesystem. A PV does not mean its backend can never fail or that it survives backend destruction or every deletion policy.

StorageClass, topology, and delayed binding belong to INV-026; CSI to INV-027; StatefulSet and stable identity to INV-028. Syntax, phases, access modes, and reclaim policy remain outside this investigation.

Engineering Reflection

Timeless Engineering Principle

Resources with different required lifetimes need independent ownership.

Architectural Honesty

Stay stateless

When another authority can reproduce every needed byte, persistence may add cost without protecting value.

Accept persistence

When correctness depends on accumulated state, disposable ownership is unacceptable.

Costs Accepted

Provisioning

Storage must exist before use.

Coordination

Attachment and reclamation need ownership.

Placement

Location constrains execution.

Ambiguity

Slow, unreachable, and failed are different.

Investigation Exercise

Trace ownership before choosing technology.

Prediction

Predict availability after replacement and rescheduling.

Experiment

Compare Pod, node, application, and platform ownership.

Observation

Separate physical existence from accessibility.

Reflection

Identify the bounded contract earned.

o o o
Run the synthesis trace after writing your prediction.

Bridge to INV-026

Storage has its own lifecycle.
But it still has a location.

The Pod no longer owns its records.

The platform owns provisioning, attachment, and reclamation.

The logical resource remains after execution stops.

Physical storage remains subject to infrastructure constraints.

Compute placement and storage topology now collide.

A replacement Pod starts on a new node while its data remains elsewhere, opening the storage placement question

Next Investigation

INV-026 - The Storage Placement Problem

How can compute placement account for storage topology?

Intellectual Lineage

This investigation inherits reconciliation and independent ownership from control theory and the architectural lineage of Borg and Omega. It does not claim they introduced Kubernetes persistent-volume APIs.

Deliberate Simplifications Ledger

Persistent storage resource semanticsStorage Resource Semantics (unassigned)
Storage placement and topologyINV-026
Stable storage interfaceINV-027
Stable identity and per-participant storageINV-028
Failure detection, recovery, and access safetyStorage Failure and Access Safety (unassigned)

Sources