Node.js Graceful-Shutdown Review for Offshore API Work

Build one shutdown timeline

Capture one timestamped lifecycle from signal delivery to process exit. On SIGTERM, mark the instance unavailable through the actual readiness mechanism, stop accepting new connections, and track every request or job already admitted. HTTP keep-alive, upgraded connections, database transactions, queue consumers, telemetry exporters, and worker threads need separate closure rules. A fixed sleep is not drainage evidence. Inject slow requests, a hung dependency, a streaming response, an active transaction, an idle keep-alive socket, and a job between checkpoint and acknowledgement. Define which work may finish, which client receives a retryable refusal, which operation must reconcile later, and what happens at the platform deadline. Ensure signal handlers cannot leave the event loop alive forever and that a second signal has an intentional meaning. The service owner defines safety; the developer instruments and tests the bounded sequence.

Use one clock for the signal, readiness transition, listener close, last accepted request, completed response, queue acknowledgement, resource close, telemetry flush, and process exit. Without that timeline, a clean exit code can hide abandoned work. Run the fixture under the same termination grace and routing behavior used by the target platform.

Separate admission from completion

Stopping new work and finishing accepted work are different controls. Close the listener, mark the instance unavailable through the real readiness path, and watch existing keep-alive connections. Test a fresh request, an idle reused connection, a slow response, and a streaming response after the signal. Record what the client sees and whether another instance can safely handle a retry.

Readiness changes do not instantly erase every route to the process. Load balancers, service proxies, and established connections have their own timing. The service owner must define the acceptable refusal and drain window; the engineer measures it rather than inserting an unexplained sleep.

Buyer decision record

Scroll sideways to read every column on a small screen.

Decide the fate of background work

Correlate load-balancer removal, readiness change, socket acceptance, application request IDs, transaction completion, queue acknowledgement, and process exit in one trace. Run the fixture with HTTP/1.1 keep-alive and every other protocol actually served. Confirm the orchestrator sends the signal the code handles and that the termination grace exceeds the application deadline plus platform routing delay. A health endpoint that flips only after listener close may be too late. Conversely, declaring unready before installing bounded handlers can create needless outage. Preserve the forced-kill experiment and the exact unfinished operations it leaves, then assign reconciliation to the correct owner instead of hiding it behind a successful restart.

Pause queue intake before claiming drainage. For a job already running, record its checkpoint, external side effects, acknowledgement state, and recovery owner. Test shutdown before the first side effect, between two effects, and after completion but before acknowledgement. A generic retry is unsafe when the external action is not idempotent.

Bound resource cleanup

Database pools, telemetry exporters, worker threads, timers, and upgraded sockets can keep the event loop alive. Give each resource an owner and a deadline. Close them in an order that still allows admitted work to finish. Inject a hung close and show that the overall deadline still wins. A second termination signal should have documented behavior instead of installing another unbounded wait.

Make forced termination visible

Run one disposable case beyond the platform grace period. Preserve the operations left uncertain and verify the next instance can reconcile them. Do not mark the run successful merely because the orchestrator starts a replacement. Alert on forced kills, excessive drain time, unfinished jobs, and transactions whose outcome is unknown. Those signals point to different recovery actions. Keep the termination reason and deadline in the same trace so an operator can distinguish an application timeout from an external eviction.

Keep readiness and shutdown ordering testable

Install the signal handler before the process advertises readiness. During termination, record when readiness changes relative to listener closure and the last accepted request. If the health endpoint shares the closing listener, the platform may miss the state transition; if readiness flips too early without replacement capacity, the rollout may reduce service availability. Test the actual routing components rather than assuming one ordering works everywhere.

Start a second instance during the fixture and send requests continuously across the transition. Classify completed responses, explicit retryable refusals, resets, timeouts, and duplicated work. A graceful process exit is only one observation. The client outcome and downstream side effects determine whether the replacement was safe.

Design reconciliation before relying on retries

For each operation that can outlive a request, record an idempotency key, transaction boundary, queue acknowledgement point, or other approved reconciliation evidence. Then kill the process at that boundary and run the recovery path. If the system cannot determine whether an external side effect occurred, the handoff must assign investigation instead of automatically repeating it.

Telemetry needs its own bounded shutdown budget. Flush a synthetic final event, make the collector slow, and prove that observation cannot hold the process beyond the platform deadline. Mark missing telemetry as an evidence gap; do not reinterpret silence as successful drainage.

Release evidence

Acceptance requires new work to stop, admitted work to reach a documented outcome, resources to close, telemetry to flush within its budget, and the process to exit before forced termination. A hanging socket, uncertain side effect, acknowledged-but-unfinished job, or success exit after failed drainage is a release stop.

Package the signal configuration, platform deadline, routing observations, request and job fixtures, resource-close results, exit codes, and reconciliation steps. The platform owner approves termination settings. The service owner approves client and job outcomes. The developer may change lifecycle code and tests inside that boundary, but does not decide that an uncertain production side effect is safe. Record the previous service revision and its shutdown behavior so reviewers can distinguish a correction from a changed workload or routing condition.

Questions about assessing Philippine developers