Investigation 012 · The Ownership Problem
A Deployment is deleted.
What happens to everything it created?
A hundred objects accumulate behind a single Deployment — ReplicaSets, Pods, Secrets, ConfigMaps. Before anything can be safely cleaned up, the system must answer a harder question: how does it know what truly belongs to what?
Begin the investigation ↓Prologue
The abandoned construction site
Cranes, generators, cement mixers, scaffolding, temporary offices, electrical cables, stacks of unused materials. Some brought in specifically for this building. Others belonging to contractors on nearby projects. Some already served their purpose. Others still needed.
What can be safely removed? Remove too little, and an abandoned site keeps consuming space and money. Remove too much, and you destroy equipment someone still depends on.
Humans answer this with context — who hired which contractor, what was rented for this project. Computers possess none of this understanding. To a computer, every object is simply another record.
First Principles
Ownership cannot be inferred. It must become explicit state.
Imagine a system that stores objects with no relationships at all. Each object has its own identity; each can be created, updated, deleted. Wonderfully simple — until the first question arrives: why does Object B exist?
The system has no answer. A user might have created it. Another object might have created it. From the system’s perspective, every possibility looks identical — there is no evidence, only objects.
A reasonable engineer might argue relationships can be inferred: if a Pod appears right after a Deployment, surely it belongs to that Deployment. But two users can create similarly-named objects independently; objects get recreated after failures; resources get restored from backup; retries reorder creation. Every heuristic starts producing wrong conclusions.
Every heuristic starts producing wrong conclusions.
Naive Architecture
The Ownership Chain
Ownership isn’t a single link — it’s a chain. A Pod exists because of a ReplicaSet, which exists because of a Deployment, which exists because of an Application.
The Pod does not directly belong to the Application — its ownership is indirect. The system must understand the entire chain, because if one link breaks, the chain leaves orphans: objects that still exist but can no longer be traced back to an intended owner.
Lab — Break a Link in the Chain
Remove the connection between the Deployment and ReplicaSet, and watch what happens to everything below it.
Chain intact: every object traces back to the Application.
Ownership is transitive. A system must preserve the entire chain, not merely individual relationships.
Ownership is transitive.
The Architecture That Almost Worked
Why store what can be inferred?
Perhaps ownership never needs to be recorded at all. Controllers already know what they create. Whenever ownership needs to be determined, just reconstruct the relationship from the current state of the system.
The design is attractive: minimal stored state, no need to update multiple objects when ownership changes, relationships never go stale because they’re always freshly reconstructed. Why store what appears derivable?
The question is whether reality is always that simple. It isn’t — and the next three experiments prove it.
It isn’t — and the next three experiments prove it.
Breaking Our Design
Every inference mechanism fails the same way
Objects are recreated. Controllers restart. Names are reused. Operations are retried. Under these conditions, inference stops being certainty — it becomes guesswork. Three increasingly hostile experiments attack three different assumptions.
Episode 1 — Names Are Not Ownership
A Deployment named frontend creates a ReplicaSet whose name embeds it; Pods follow the same pattern. The hierarchy looks obvious. But the Deployment is deleted, and later another Deployment is created with exactly the same name.
To a human these are different objects. To a system relying only on names, they look identical. Names describe identity to humans — they can be reused, changed, chosen independently. Nothing about a name proves ownership.
Episode 2 — Time Is Not Ownership
Causes usually precede effects: Deployment created, then ReplicaSet, then Pods a moment later. Surely the timeline tells the story. But controllers retry, objects get restored from backup, requests get delayed, components restart.
Two controllers examining the same cluster may observe completely different event sequences — neither is wrong, they simply saw different histories. Time records chronology. It does not record intent.
Episode 3 — Labels Are Not Ownership
A Deployment, a Service, and a NetworkPolicy can all select the exact same label app=frontend. Do all three own the Pod? Clearly not — the Deployment manages its lifecycle, the Service routes traffic to it, the policy applies security rules to it.
The label tells us a relationship exists. It never tells us what kind. Labels describe membership; ownership describes responsibility. A Pod may match many selectors and still belong to only one lifecycle.
A Pod may match many selectors and still belong to only one lifecycle.
Lab — Test Every Inference Mechanism
Try to prove ownership using names, creation time, and labels. Watch each one collapse under a realistic scenario.
Click a mechanism to test it against reality.
Turning Point
We have exhausted every reasonable inference mechanism
Names cannot prove ownership. Time cannot prove ownership. Labels cannot prove ownership. Each described some property of an object. None described why the object existed.
If ownership cannot be reconstructed, it must be recorded.
If ownership cannot be reconstructed, it must be recorded.
The Ownership Contract
Five responsibilities every correct design must satisfy
Recorded as durable state — never inferred from naming, order, or controller memory.
Reference a stable identity, not a reusable human-readable name.
Every controller, scheduler, and garbage collector must see the same relationship.
Restarts, leader elections, and partitions must never lose ownership.
Ownership records the relationship; it does not by itself decide what action to take.
Kubernetes satisfies this contract through Owner References — an ownerReferences field in every object’s metadata:
ownerReferences:
- apiVersion: apps/v1
kind: ReplicaSet
name: frontend-rs
uid: 8f3c...
controller: trueLab — Why a UID Instead of a Name?
A Deployment named frontend is deleted, then recreated with the same name. Compare what a name-based reference sees versus a UID-based reference.
Old Pods still reference name: frontend.
New Deployment is also named frontend.
Result: pending…Old Pods reference uid: 8f3c-old.
New Deployment has uid: 71ae-new.
Result: pending…Waiting to simulate recreation.
Engineering Reflection
Shared state produces shared decisions
A distributed system succeeds when independent components reach the same conclusion without directly coordinating. That’s only possible when they observe the same information. If every controller reconstructed ownership independently, different controllers could reach different conclusions about the same object.
Ownership is not lifecycle. Ownership answers “who is responsible for this object?” Lifecycle answers “what should happen when that responsibility changes?” Recording ownership does not automatically imply deletion — it merely provides the information required to make lifecycle decisions safely.
Costs Accepted
Additional metadata. Every dependent object carries an ownerReferences entry, however small.
Relationship maintenance. Controllers must record ownership whenever they create new resources.
Storage and bandwidth. Ownership information occupies space in every object's state.
These costs are intentional — significantly smaller than the operational cost of incorrectly deleting resources, leaking abandoned objects, or requiring humans to reconstruct ownership manually.
Investigation Exercise
Exercise 1 — Can you infer ownership? Given a Deployment, ReplicaSet, and Pods with matching names, determine ownership without any explicit metadata. Then delete the Deployment and recreate one with the same name. Can you still determine ownership with certainty?
Exercise 2 — Does time reveal ownership? Given a creation timeline, ask whether every observer can reconstruct the same relationships once retries, backups, and restarts are introduced.
Exercise 3 — Are labels ownership? Given a Deployment, Service, and NetworkPolicy all selecting the same label, determine which one owns the Pods — and what information is still missing.
Bridge to INV·013
Knowing who owns an object is not the same as knowing what happens next
We now record ownership as explicit, durable, identity-based state. Every component can answer “who is responsible for this object?” consistently.
But suppose the Deployment at the top of the chain is deleted. Should the ReplicaSet disappear immediately? Should the Pods go with it? Top-down or bottom-up? What if deletion is only partially complete?
The next investigation explores how a distributed system safely removes resources without deleting too much, too little, or leaving the system inconsistent.
Deliberate Simplifications Ledger
| We deliberately postponed | Owned by |
|---|---|
| How Kubernetes safely removes dependent objects after an owner is deleted | INV·013 — The Garbage Collection Problem |
| Multiple owner references on a single object (shared ownership) | Implementation detail — ownerReferences slice semantics |
| Cross-namespace ownership restrictions | Implementation detail — API server enforcement |
The controller field identifying the managing owner among multiple owners | Implementation detail — ownerReferences[].controller |
The blockOwnerDeletion field and foreground deletion propagation | INV·013 — The Garbage Collection Problem |
Sources
Official Documentation: Kubernetes Documentation — Owners and Dependents; Kubernetes API Conventions — ownerReferences field specification; Kubernetes Garbage Collection design documentation.
Source Code: k8s.io/apimachinery/pkg/apis/meta/v1 — OwnerReference struct definition.
Next: INV·013 — The Garbage Collection Problem