StatefulSets

# 1) When to use a StatefulSet
# • Each pod has identity (replica id, leader/follower)
# • Each pod has its own persistent storage
# • Pods must start / scale up in order
# • Network DNS name must be stable across restarts
#
# Use Deployment for stateless web servers. Use StatefulSet for Postgres, Kafka, Redis Cluster, Elasticsearch.

# 2) A complete example — a 3-node cluster with its own storage
apiVersion: v1
kind: Service
metadata:
    name: web
    labels: { app: web }
spec:
    clusterIP: None                    # 'headless' — required for stable per-pod DNS
    ports:
        - port: 80
          name: http
    selector:
        app: web
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
    name: web
spec:
    serviceName: web                    # MUST match the headless Service name
    replicas: 3
    selector:
        matchLabels: { app: web }
    template:
        metadata:
            labels: { app: web }
        spec:
            containers:
                - name: web
                    image: nginx:1.27-alpine
                    ports:
                        - containerPort: 80
                            name: http
                    volumeMounts:
                        - name: data
                            mountPath: /usr/share/nginx/html
            terminationGracePeriodSeconds: 30
    volumeClaimTemplates:
        - metadata:
              name: data
          spec:
              accessModes: ['ReadWriteOnce']
              storageClassName: gp3
              resources:
                  requests:
                      storage: 10Gi

# 3) What this produces
#   pods:           web-0, web-1, web-2
#   pvcs:           data-web-0, data-web-1, data-web-2
#   dns:            web-0.web.<ns>.svc.cluster.local, web-1.web.<ns>.svc.cluster.local, …
#   pod ordering:   web-0 must be Ready before web-1 starts

# 4) Inspect
kubectl get statefulset,svc,pod,pvc -l app=web
kubectl rollout status statefulset/web
kubectl describe statefulset web

# 5) DNS — every pod gets a name
# Inside the cluster (any pod):
nslookup web-0.web.default.svc.cluster.local        # → web-0's IP
nslookup web.default.svc.cluster.local               # → all pod IPs (headless service)
# This lets distributed systems address peers by name (e.g. Cassandra seeds, Kafka brokers, Postgres replicas).

# 6) Scaling
kubectl scale sts/web --replicas=5      # starts web-3, then web-4 (in order)
kubectl scale sts/web --replicas=2      # terminates web-4 first, then web-3 (reverse order)
# Scaling down does NOT delete the PVCs by default — see persistentVolumeClaimRetentionPolicy below.

# 7) Rolling update strategy
spec:
    updateStrategy:
        type: RollingUpdate
        rollingUpdate:
            partition: 0                # update pods with ordinal >= partition; 0 = all
            maxUnavailable: 1            # allow 1 pod down at a time (KEP-961, 1.27+)

# Canary pattern with partition:
kubectl patch sts/web -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":2}}}}'
kubectl set image sts/web web=myrepo/web:1.1.0     # only web-2 updates; web-0, web-1 stay on old
# Watch web-2 health, then drop partition to 0 to roll forward.

# 8) OnDelete strategy — manual updates
spec:
    updateStrategy:
        type: OnDelete
# Now editing the template doesn't auto-roll; you delete pods and the new spec is applied as they come back.
# Useful for very sensitive workloads where you choreograph upgrades.

# 9) PVC retention policy (Kubernetes 1.27+)
spec:
    persistentVolumeClaimRetentionPolicy:
        whenDeleted: Retain         # delete the StatefulSet → keep PVCs (default for safety)
        whenScaled:  Delete         # scale down → delete the PVCs for removed ordinals
# Default = Retain for both. Set whenScaled=Delete to reclaim disk on downscale.

# 10) Probes — slow-starting databases need wide windows
livenessProbe:
    exec:
        command: ['pg_isready', '-U', 'postgres']
    initialDelaySeconds: 30
    periodSeconds: 10
readinessProbe:
    exec:
        command: ['pg_isready', '-U', 'postgres']
    initialDelaySeconds: 5
    periodSeconds: 5
startupProbe:
    exec:
        command: ['pg_isready', '-U', 'postgres']
    failureThreshold: 60
    periodSeconds: 5

# 11) Pod identity and topology
spec:
    template:
        spec:
            affinity:
                podAntiAffinity:
                    requiredDuringSchedulingIgnoredDuringExecution:
                        - labelSelector: { matchLabels: { app: web } }
                          topologyKey: kubernetes.io/hostname
            topologySpreadConstraints:
                - maxSkew: 1
                    topologyKey: topology.kubernetes.io/zone
                    whenUnsatisfiable: ScheduleAnyway
                    labelSelector: { matchLabels: { app: web } }
# Don't co-locate replicas on the same node or AZ — a single failure can't take down the cluster.

# 12) Operators — the right abstraction for real DBs
# Hand-rolling Postgres on a StatefulSet is a learning exercise; in production use:
#   • CloudNativePG / Crunchy / Zalando Postgres Operator
#   • Strimzi for Kafka
#   • Redis Operator (multiple options)
# Operators wrap the StatefulSet + Services + secrets + backups + failover in a CRD.

# 13) Working with the storage
kubectl get pvc -l app=web
# data-web-0   Bound   pvc-…   10Gi   RWO   gp3
# Each PVC is bound to a specific PV; deleting the pod KEEPS the PVC; rescheduling re-attaches.
# Replacing a node: the PVC re-binds when the pod reschedules in the same AZ.

# 14) Pod disruption budget — protect during voluntary disruption (drains, upgrades)
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
    name: web
spec:
    minAvailable: 2
    selector:
        matchLabels: { app: web }
# Drains will wait if respecting the PDB would push availability below 2.

# 15) Backups — your responsibility
# StatefulSets manage pods and PVCs. They DO NOT back up your data.
# Pair with:
#   • Velero or Veeam for PVC + cluster object snapshots
#   • App-aware backups (pg_basebackup, kafka-mirror, redis BGSAVE)
#   • Cross-region copies for DR

# 16) Migrating to a new cluster
# • Take consistent app-level backup
# • Restore in new cluster
# • Switch DNS / Service / Ingress
# • Do NOT try to migrate live PVCs by hand — that path is full of subtle bugs

# 17) StatefulSet vs Deployment vs DaemonSet
# Deployment   — stateless, identical pods, can scale to 0, fast rollouts
# StatefulSet  — ordered, named, per-pod storage
# DaemonSet    — one pod per node (logging agents, kube-proxy)
# Job/CronJob  — run-to-completion (batch, scheduled tasks)

# 18) Common bugs
# • Forgot the headless Service or its name mismatches serviceName — pods have no stable DNS
# • PVC stuck in Pending — no StorageClass / no PV provisioner / wrong access mode
# • Pod scheduled in wrong AZ → can't bind to PV from another AZ — set topology requirements
# • Scale down deleted data — set persistentVolumeClaimRetentionPolicy.whenScaled=Retain
# • Pod-0 stuck in Init — readiness probe too tight; ordered start blocks 1,2,3,…
# • Manual hand-rolled DB on StatefulSet without operator → 3am pages about failover
# • Forgot PodDisruptionBudget → node drain takes everyone down at once

# For pods that need stable identities (db-0, db-1) + sticky storage.