3. Deployments vs StatefulSets vs DaemonSets: When to Use Each

Every Kubernetes workload controller answers the same question, and the docs never say it out loud: are your replicas interchangeable, or do they have identity?

That's it. That's the whole decision. Deployments are for interchangeable replicas. StatefulSets are for replicas with identity. DaemonSets are for "one per node, no more, no less." Everything else — the YAML differences, the naming schemes, the storage behavior — falls out of that one distinction.

I wish someone had said this to me before I spent an evening reading the StatefulSet docs trying to figure out whether our Postgres "counts as stateful." (Spoiler: state and StatefulSet are related but not the same thing, and yes, that naming is a menace.)

In How Kubernetes Actually Thinks I made the case that you never create bare Pods — you declare a desired state and a controller maintains it. This post is about picking which controller. We'll need the answer immediately because next, we'll deploy Relay's API as a Deployment, and in the persistent storage post I put Postgres on the cluster and make a choice some people will yell at me for.

The mental model: cattle vs pets, but for real this time

You've heard the cattle/pets analogy. Here's what it concretely means in Kubernetes terms.

A Deployment treats every replica as cattle. When you run three replicas of relay-api, the pods get names like relay-api-7d9f8b6c4-x2vlp — a random hash, then more random characters. Kubernetes is telling you something with that name: this pod's individual existence is meaningless. Kill one, and the ReplicaSet spins up a replacement with a completely different name, possibly on a different node, with a different IP. No pod is special. Traffic reaches them through a Service that load-balances across whoever happens to be alive.

A StatefulSet treats every replica as a pet with a name tag. Pods are named postgres-0, postgres-1, postgres-2 — stable, ordinal, predictable. If postgres-1 dies, its replacement is also named postgres-1, gets the same stable DNS hostname (postgres-1.postgres.default.svc.cluster.local), and — this is the important part — reattaches to the same PersistentVolumeClaim the old postgres-1 was using. Identity survives death.

Three things come bundled with that identity:

Stable network names. Each pod gets its own DNS record via a headless Service. Peers can find postgres-0 specifically, not just "some postgres pod."
Per-replica storage. volumeClaimTemplates stamps out one PVC per pod — data-postgres-0, data-postgres-1 — and each pod always reconnects to its own.
Ordered operations. Pods start in order (0, then 1, then 2), and terminate in reverse. Rolling updates go highest-ordinal-first.

Ask yourself: who actually needs all that? Clustered databases. Kafka brokers. Anything where replica 0 is a primary and replica 1 is a follower that replicates from replica 0 by name. The identity machinery exists so that members of a distributed system can find each other and keep their own data across restarts.

Now the trap, because everyone falls into it: "my app has state" does not mean "my app needs a StatefulSet." A single-replica database has state, obviously — but it has no peers. There's nobody who needs to find postgres-0 by ordinal, no startup ordering to enforce, no per-replica claim fan-out (there's one replica). The StatefulSet's entire feature set is about coordinating multiple stateful members. With one member, you're paying StatefulSet's operational quirks for features you're not using.

Deployments: the default, and the YAML you'll write most

A Deployment doesn't manage pods directly. It manages ReplicaSets, and the ReplicaSet manages pods. That indirection is what makes rolling updates work: when you change the pod template (new image, new env var), the Deployment creates a new ReplicaSet and gradually shifts replicas from old to new. Rollback is just shifting back. We'll watch that happen live in the multi-replica scaling post.

Here's Relay's API, minimally:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: relay-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: relay-api
  template:
    metadata:
      labels:
        app: relay-api
    spec:
      containers:
        - name: relay-api
          image: relay-api:v1
          ports:
            - containerPort: 3000

The selector / labels pairing confused me at first — why say app: relay-api twice? Because the Deployment finds its pods by label query, not by parentage. The selector is the search; the template labels make new pods findable by that search. They must match or the API server rejects it.

One field people ignore until it bites them: strategy. The default is RollingUpdate, which keeps old and new pods running simultaneously during a rollout. That's exactly right for stateless APIs and exactly wrong for some other things — hold that thought.

Opinion: Deployment is the default. Not "a" default — the default. Stateless API? Deployment. Worker consuming a queue? Deployment. Frontend? Deployment. You should need a specific, nameable reason to reach for anything else. In a typical cluster, Deployments outnumber StatefulSets ten to one, and that ratio is healthy.

StatefulSets: identity, and what it costs

Here's what a real multi-replica StatefulSet looks like, so you can see the machinery:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres          # headless Service that owns the per-pod DNS
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:          # one PVC stamped out PER POD
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 2Gi

Two structural differences from the Deployment. serviceName points at a headless Service, which is what gives each pod its stable DNS record. And volumeClaimTemplates replaces a normal volume reference — Kubernetes creates data-postgres-0, data-postgres-1, data-postgres-2, and each pod is permanently married to its own claim.

Note what this YAML does not do: it does not make those three Postgres instances replicate. Kubernetes gives you stable names and stable storage; the actual primary/follower configuration, failover, and replication wiring are entirely your problem. This is the second big misconception — people think replicas: 3 on a Postgres StatefulSet gives them a highly-available database. It gives them three independent databases with tidy names. The clustering logic has to come from somewhere else, which is exactly why database operators (CloudNativePG, Zalando's postgres-operator) exist: they're the brain that turns "three pods with identity" into "one replicated database."

The costs of identity, since nothing is free: deleting a StatefulSet doesn't delete its PVCs (safety feature, surprises everyone), a pod stuck in a bad state can block the whole ordered rollout behind it, and scaling down doesn't clean up the departed replica's data. StatefulSets are heavier to operate. That's fine when you need them. It's dead weight when you don't.

DaemonSets: one per node, by definition

DaemonSets answer a different question entirely. Not "how many replicas?" but "is it on every node?" You don't set a replica count — the node count is the replica count. Add a node to the cluster, the DaemonSet schedules a pod onto it automatically. Remove the node, the pod goes with it.

The use cases are all infrastructure-shaped: log collectors, node monitoring agents, CNI networking plugins, storage daemons. Things that need to exist at every node because they interact with the node itself. We'll actually deploy one in the logging and observability post — Grafana Alloy runs as a DaemonSet so every node has a local agent shipping that node's container logs.

If you're writing application code, you almost certainly don't want a DaemonSet. I mention it here mostly so that when you run kubectl get pods -A on your k3d cluster and see system pods replicated exactly once per node, you know what's doing that.

And for completeness, the fourth shape: Jobs run pods to completion — the pod is supposed to exit, and exiting successfully is the desired state, which inverts everything the other controllers assume. CronJobs create Jobs on a schedule. Relay's database backups will be a CronJob in the disaster recovery post.

The controversial part: Relay's Postgres is a Deployment

In the persistent storage post, Relay's Postgres goes onto the cluster like this: a plain Deployment, replicas: 1, strategy: Recreate, one ordinary PVC. Not a StatefulSet.

Some people consider this heresy. Here's the actual reasoning.

At replicas: 1, every StatefulSet feature is inert. Stable ordinal identity? There's one pod; a regular PVC reference gives it "the same storage every time" just fine. Per-replica claim templates? One replica, one claim. Ordered startup? You can't order one thing. Peer discovery? No peers. The StatefulSet would be pure ceremony.

The one thing I do have to handle is the thing StatefulSets handle implicitly: never letting two Postgres pods mount the same data directory simultaneously. A default RollingUpdate Deployment would try exactly that during an update — start the new pod before killing the old one. Two postgres processes fighting over one data directory is a corrupted database. Hence:

spec:
  replicas: 1
  strategy:
    type: Recreate   # kill the old pod fully before starting the new one

Recreate means updates cause a few seconds of database downtime. For a learning project — honestly, for plenty of small production systems — that's a fair trade, and I'd rather take it explicitly than get accidental corruption politely.

Exactly when this stops being okay: the moment you want any replication. A read replica, a hot standby, anything where a second Postgres instance exists — you now have peers that need identity, and you've left Deployment territory. But here's my stronger opinion: you shouldn't graduate to a hand-rolled StatefulSet either. Configuring Postgres replication and failover yourself, on top of raw StatefulSet primitives, is a part-time job with a pager attached. At that point use an operator (CloudNativePG is the current best answer) or pay for a managed database and let it be someone else's pager. The hand-written StatefulSet-for-Postgres occupies a weird middle zone: too complex to be simple, too naive to be safe.

Single replica: Deployment + Recreate, accept the downtime window. Real HA: operator or managed. The DIY StatefulSet in between is where the horror stories live. I'll audit this exact single-point-of-failure honestly in the production readiness post.

The decision, compressed

Replicas interchangeable? Deployment. This is 90% of everything you'll deploy, including all of Relay's API tier.
Replicas need to find each other by name and keep per-replica data? StatefulSet — but if it's a database, strongly consider an operator or managed service instead of raw YAML.
Exactly one per node, because it services the node? DaemonSet.
Runs and finishes? Job, or CronJob on a timer.
Has state but only one replica? Trick question — that's still a Deployment. State lives in the PVC, not the controller. Just use Recreate.

Next up, we stop theorizing: getting Kubernetes running locally, where the first Deployment of relay-api goes onto a real cluster and immediately greets me with ImagePullBackOff.