Kubernetes Readiness-Gate Review for Offshore Release Support

Decide what the extra condition protects

Name the custom PodCondition exactly and identify the controller permitted to patch status. New conditions default to false for readiness calculation until written, so controller absence can halt a rollout even when containers are healthy. Rehearse Pod creation before external registration, successful registration, rejected registration, slow external propagation, controller restart, deleted external object, and Pod deletion during an update. Observe Pod conditions, EndpointSlice membership, Deployment availability, external target state, events, and client probes on one timeline. Readiness does not drain every protocol instantly and does not prove application correctness. Define a timeout and recovery owner without adding a second controller that races the first. Verify RBAC is scoped to the necessary status update and that cleanup cannot remove another Pod’s registration. Platform owners approve production controllers and emergency bypass; the engineer provides a repeatable fixture and rollback manifest.

A readiness gate is useful only when container health is insufficient to decide whether a Pod should receive traffic. Name the external dependency and the exact state that makes the Pod eligible. Examples might include registration with an external load balancer or completion of a separate provisioning step, but the review must use the system that actually exists. If a normal readiness probe can test the condition safely, adding a controller and custom status writer may create more failure modes than it removes.

Trace the condition from Pod creation

Create a Pod with the gate configured and watch its conditions before the custom controller writes anything. Record the condition type, status, reason, message, transition time, and observed generation. Compare those values with container readiness, the Pod Ready condition, Deployment availability, and EndpointSlice membership. This timeline answers a practical question: did traffic remain withheld for the intended reason, or did another probe or rollout rule determine the outcome?

Repeat the trace when the external registration completes quickly, completes slowly, and fails. The controller must distinguish an unfinished operation from a confirmed rejection. An empty status, an old status copied from another generation, and an explicit false status have different diagnostic meaning. Preserve the resource versions used for each patch so an update conflict is visible rather than mistaken for a controller outage.

Buyer decision record

Scroll sideways to read every column on a small screen.

Break the controller without losing the rollout

Inspect status conditions with observed generation, reason, message, and transition time rather than reducing evidence to one Ready boolean. Compare the custom controller’s work queue with API updates and external target registration. Introduce an update conflict and temporary API failure to verify retry behavior without duplicate external resources. During rollout, confirm maxUnavailable and maxSurge interact safely with the delayed readiness condition. Pod deletion needs a finalization strategy that cannot hang forever; document its timeout and orphan cleanup owner. A manual status patch is diagnostic evidence only, not a routine recovery plan. Keep controller logs, resource versions, endpoint snapshots, and synthetic client results together.

Stop the controller after Pod creation but before it writes the condition. Restart it and verify that queued work is reconstructed from cluster state instead of relying on memory from the former process. Then introduce a temporary API write failure and an external API timeout. Count retries and external objects. A retry that creates a second registration is not recovery, even if one of the registrations eventually becomes healthy.

Test replacement capacity

Delayed readiness consumes rollout capacity. Run the fixture with the Deployment's actual maxSurge and maxUnavailable values and note when old replicas terminate. If all replacement Pods wait on the same external service, a controller slowdown can stall the rollout or reduce available capacity. The evidence should show replica counts and endpoint membership throughout the update. Do not infer customer availability from the final Deployment status alone.

Test the opposite failure as well: the controller reports true before the external target can serve a request. Send a synthetic request through the external path and compare its time with the condition transition. The controller should not predict completion. If the external system has eventual propagation, define the observation that closes that gap and who owns its timeout.

Make deletion specific and bounded

Delete one Pod while its external registration exists. The cleanup path must identify that Pod's resource without selecting by a mutable display name or a broad label. Verify that another replica remains registered. If a finalizer is used, make its timeout and orphan procedure explicit because a permanently unavailable external API can otherwise leave Pods stuck in termination.

Now delete a Pod before registration finishes and during a controller restart. Record whether the pending operation is cancelled, completed and then removed, or left for an orphan reconciler. Each outcome can be reasonable if it is deliberate. The dangerous outcome is an external target that survives unnoticed and later points at a reused address.

Keep emergency action under platform ownership

A manual status patch can prove that the gate is responsible for a stalled rollout, but it is not a standing recovery method. State who may patch status, how the external registration is checked first, and when a rollout must pause instead. An emergency bypass needs a narrow workload, a time limit, observation of the traffic path, and a follow-up action that restores controller ownership.

Review RBAC separately from functional behavior. The controller should patch only the status it owns on the intended Pods. It should not receive broad workload mutation merely because that makes development easier. Capture the service account, role rules, binding scope, and a denied operation outside that scope.

Evidence for the release decision

Pass when external registration and Pod conditions transition together, EndpointSlices reflect the intended state, controller restart recovers, and deletion cleans only the correct resource. Indefinite false conditions, excess RBAC, racing writers, or a rollout with no bounded escape remain platform-owner blockers.

The review packet includes the condition name, controller revision, RBAC, external resource identifier, Pod and EndpointSlice timelines, synthetic request results, failure injections, deletion outcomes, alert rules, and rollback manifest. Platform owners decide whether the controller and bypass procedure are acceptable. Service owners confirm what traffic readiness means. The developer supplies the reconciler change and repeatable evidence without using a production status patch as proof.

Questions about assessing Philippine developers