A Kubernetes StatefulSet Partitioned-Rollout Study for Ordered Services

Key Stats

• 1 version-pinned Kubernetes cluster
• 3 stable Pod ordinals
• 16 rollout, failure, and recovery cases

Key Takeaways

• Use the partition as an explicit ordinal boundary.
• Verify identity and storage separately from readiness.
• Rehearse rollback before changing the next ordinal.

The rollout decision

This study asks how a team should stage one StatefulSet revision when Pod identity and update order matter. A synthetic three-replica service exposes its ordinal, controller revision, persistent-volume marker, readiness state, and application compatibility result. The operator raises and lowers rollingUpdate.partition to choose which ordinals may change. The fixture pauses after each boundary, injects a failure into the first updated Pod, and records what Kubernetes does before anyone approves the next ordinal.

The result is a rollout rule for one ordered service, not a claim that StatefulSet partitions are a universal canary system. A partition controls which ordinal numbers receive an update under the RollingUpdate strategy. It does not decide whether the new application is semantically compatible, whether stored data can be read by an older revision, or whether rollback is safe. Those decisions stay with the application and data owners. The engineer supplies observable evidence about controller behavior, identity, storage, and readiness.

Documented behavior to test

Kubernetes documents that StatefulSet Pods have stable ordinal identities and are created and deleted in a defined order under the default OrderedReady policy. With RollingUpdate, the controller updates Pods in reverse ordinal order, waiting for an updated Pod to become Running and Ready before proceeding. A partition causes Pods with an ordinal lower than the partition to remain at the previous template while Pods at or above it receive the new template. The fixture tests those rules on the pinned cluster rather than assuming every workload is configured for them.

The API reference defines updateStrategy, rollingUpdate.partition, maxUnavailable where supported, podManagementPolicy, currentRevision, updateRevision, currentReplicas, updatedReplicas, readyReplicas, and availableReplicas. Capture the exact API server version and feature gates before interpreting those fields. Keep maxUnavailable at its declared baseline unless it is the variable under study. A percentage or parallel policy would change the experiment. The manifest, not a familiar mental model, determines which guarantees apply.

Build a stateful but harmless fixture

Create a headless Service and a StatefulSet named ledger with three replicas: ledger-0, ledger-1, and ledger-2. Each Pod mounts its own test volume, writes a persistent marker containing its ordinal and a generated fixture identifier, then serves a small status endpoint. The endpoint reports image revision, ordinal, marker hash, readiness, and a synthetic protocol version. It contains no credentials or production data. The old and new images differ only in revision label, compatibility behavior, and the seeded readiness control.

Pin cluster, kubectl, storage driver, container image digests, manifest hash, namespace, and observation commands. Record StatefulSet generation, observedGeneration, currentRevision, updateRevision, Pod UID, creation time, node, image ID, readiness transitions, volume claim name, and marker hash. A Pod recreation should change UID while preserving the expected ordinal, claim, and marker. A rollout conclusion based only on Pod names misses whether the controller actually replaced a Pod or whether storage identity followed it correctly.

Establish the old revision

Apply revision A with partition 3, wait for all three Pods to become Ready, and verify every volume marker. Query the synthetic protocol from each ordinal directly and through the governing Service. Save the controller revisions and status fields. Delete ledger-1 once before the rollout and prove that its replacement keeps the ledger-1 network identity and claim marker. This baseline separates ordinary StatefulSet replacement from the later template update.

Seed negative controls before changing the template. A wrong marker must fail the identity check. A status endpoint that reports revision B while the container image remains A must fail the revision check. A Pod that is Running but not Ready must not count as an accepted stage. These controls matter because a green kubectl wait can otherwise conceal an incorrect assertion or a fixture that reads a mutable label rather than the running image and mounted state.

Update only the highest ordinal

Change the template to revision B while keeping partition 3. The updateRevision should change, but no Pod should move to B because every ordinal is below the partition boundary. Then set partition 2. The controller should replace ledger-2 while ledger-1 and ledger-0 remain on A. Record the deletion and creation sequence, readiness, revision labels, endpoint result, claim name, and marker. Do not lower the partition merely because the new Pod reaches Running.

Acceptance for this stage requires ledger-2 to report B, retain its expected persistent marker, pass the compatibility probe, and remain Ready for the declared observation window. The two lower ordinals must still report A and serve their expected protocol. Send synthetic reads and writes across the mixed revision set to expose a compatibility assumption. The application owner defines the allowed mixed-version behavior. Kubernetes can order replacement, but it cannot certify that A and B understand the same stored or network data.

Inject a readiness failure

Repeat the ledger-2 stage with a revision B variant whose readiness endpoint fails after startup. Observe the controller status, Pod events, restart behavior, and whether lower ordinals remain unchanged. The expected safety property is that the ordered rollout does not advance to ledger-1 while ledger-2 is not Ready. Preserve the exact timeline and distinguish container restarts from Pod replacement. A liveness probe is not added unless the experiment explicitly studies it, because liveness could erase the evidence by creating a separate restart loop.

Repair the readiness configuration without lowering the partition. Confirm that ledger-2 eventually becomes Ready with the intended revision and the same claim marker. If the controller appears stuck after reverting the template, test the documented forced-rollback caveat on the pinned version: an unhealthy Pod created under a bad template may require manual deletion after the template is reverted. Record whether manual deletion was needed, who authorized it, the old and new UIDs, and the storage marker after recreation.

Advance and pause at each boundary

After ledger-2 passes, lower the partition to 1. Verify that only ledger-1 changes, then repeat identity, storage, readiness, protocol, and mixed-version checks across B at ordinals 2 and 1 with A at ordinal 0. Lower the partition to 0 only after the second review is accepted. This sequence gives the reviewer an explicit stop before each ordinal. It does not create an automatic approval merely because the controller is capable of continuing.

At every stage, compare desired replicas, current replicas, updated replicas, ready replicas, currentRevision, and updateRevision with per-Pod observations. Status counts can lag or summarize a state that is too coarse for the decision. Preserve watch output or timestamped snapshots rather than one final describe command. A pass requires agreement between controller status, Pod identity, image digest, readiness, application response, and volume marker. Disagreement pauses the rollout even when aggregate availability looks healthy.

Rehearse rollback from a mixed revision

Stop with ledger-2 and ledger-1 on B and ledger-0 on A. Restore the revision A template while holding a partition that changes only the intended ordinal. Observe updateRevision and the reverse-ordinal replacement sequence. Verify that each recreated Pod can read its existing marker and serve the old synthetic protocol. If revision B wrote a format that A cannot read, the application compatibility probe must fail even though Kubernetes successfully applies the older template.

Test rollback with one Pod unavailable, one pending because of a seeded scheduling constraint, and one readiness failure. Keep these cases separate so the evidence identifies the blocking condition. Do not force-delete a Pod, remove a finalizer, detach storage, or broaden scheduling permissions merely to make the rehearsal finish. Those operations have different risk owners. The handoff names the smallest authorized recovery action and the evidence required before moving the partition again.

Check configuration variants without mixing claims

Run a separate comparison with Parallel podManagementPolicy only if the service uses it. Document which ordering expectations no longer apply to scaling and initial management, and retain the rollout observations independently. If the cluster and workload use maxUnavailable for StatefulSet rolling updates, test the exact configured value in another lane. Do not combine partition, availability, scheduling, and storage changes in one run, because a successful outcome would not reveal which setting controlled it.

Also test a no-op template application, a rapid second template change before the first partition stage completes, and an operator who accidentally lowers the partition by two ordinals. Admission policy or release tooling may prevent the last case, but the fixture should expose the consequence in its approved namespace. Record controller revisions and Pod images so an intermediate revision cannot disappear from the narrative. The recovery rule must choose a known template and partition rather than guessing from whichever Pod happens to be Ready.

Operational evidence and handoff

The evidence bundle contains cluster and client versions, feature gates, namespace, manifest and image hashes, storage class, partition changes, controller revisions, per-Pod UID and image history, readiness events, claim and marker mapping, compatibility results, seeded failures, rollback results, commands, timestamps, owners, and exclusions. Another engineer reproduces the baseline replacement, partition 2 update, readiness failure, recovery, partition 1 update, and mixed-version rollback from a clean namespace.

Cluster access stays limited to the approved test scope. The developer may prepare manifests, synthetic images, checks, and a proposed rollout procedure. The platform owner controls cluster policy, storage operations, force deletion, and production rollout. The application and data owners approve mixed-version and rollback compatibility. Release approval requires a named person; a passing fixture does not grant it. The final procedure includes stop conditions, rollback commands, and the observation window for each ordinal.

Decision rule and limits

Pass requires only intended ordinals to update at each partition, stable ordinal and claim mapping, correct marker retention, detectable seeded failures, no advance past an unready Pod under the tested policy, accepted mixed-version behavior, and a reproduced rollback path. Conditional pass identifies untested storage, scheduling, or compatibility cases with an owner. Fail preserves the first unexpected replacement, identity mismatch, data incompatibility, or uncontrolled advance. The useful result is a reviewed ordinal-by-ordinal procedure for this StatefulSet.

The study does not prove application replication, database consensus, backup restoration, zone failure, storage durability, network ordering, or production capacity. A synthetic readiness endpoint can differ from real dependency health. Controller behavior may change with Kubernetes version, feature gates, podManagementPolicy, update strategy, maxUnavailable, admission, scheduler, or storage driver. Re-run after any of those change, after altering probe semantics or data format, and before relying on the procedure for another StatefulSet.

Sources checked for this study

Kubernetes StatefulSet documentation defines stable identity, ordered behavior, RollingUpdate, partitions, controller revisions, and the forced-rollback caveat. The Kubernetes API reference defines StatefulSet specification and status fields used by the fixture. These first-party sources describe controller contracts. The three-ordinal workload, persistent markers, compatibility probes, failure injection, evidence thresholds, and review sequence are DeveloperOffshore.com analysis for a bounded engineering handoff.

The live documentation can describe a newer release than an installed cluster. Record the applicable Kubernetes version and retain the checked references with the run. API availability does not prove that a cluster enables an optional feature or that a storage provider preserves the assumed behavior. Unknown capability stays unknown until a version-pinned, authorized test demonstrates it. Do not rewrite a successful controller transition as proof that the application or its data survived correctly.

Evidence table

Good distributed work is observable at the handoff: the result, evidence, limitations, and next owner are all explicit.

Frequently asked questions

Related Research

• DevOps release support
• Legacy application maintenance
• Research library