A Node.js service can be alive but unable to serve traffic. It can also be temporarily unable to serve traffic without needing a restart. Kubernetes gives us three different probes because those are three different conditions. Treating them as one is a reliable way to turn a brief dependency slowdown into a fleet-wide restart storm.
1. Give each probe exactly one job
Startup protects a slow-starting process from premature liveness checks. Readiness answers whether this particular pod should receive new requests. Liveness asks whether restarting this process is more likely to help than letting it recover. Kubernetes suppresses readiness and liveness probes until a configured startup probe succeeds. Repeated liveness failures restart the container; readiness failures remove the pod from normal Service traffic. Those are materially different operational actions.
A common mistake is making liveness query MongoDB, Redis or another remote dependency. If the dependency fails across the fleet, all replicas may fail liveness and restart simultaneously. Restarting them cannot repair the database. It adds startup load, reduces capacity and makes the incident harder to debug. Keep liveness cheap and focused on unrecoverable local failure. A liveness endpoint returning HTTP 200 proves only that its handler responded; it is not proof that the whole service is healthy.
2. Implement explicit endpoints in Node.js
Here is a minimal Express-style example. Replace the readiness predicate with application-specific checks. Do not expose sensitive diagnostics or credentials in health responses.
import express from 'express';
const app = express();
let initialized = false;
let draining = false;
// Set initialized after essential local startup work finishes.
app.get('/health/startup', (_req, res) =>
res.sendStatus(initialized ? 200 : 503));
app.get('/health/live', (_req, res) =>
res.sendStatus(200));
app.get('/health/ready', (_req, res) =>
res.sendStatus(initialized && !draining ? 200 : 503));
const server = app.listen(3000, async () => {
// Load local configuration and warm required resources.
initialized = true;
});
process.once('SIGTERM', () => {
draining = true;
server.close((err) => {
process.exitCode = err ? 1 : 0;
});
});
This is a teaching example, not a drop-in shutdown framework. Production shutdown must bound the drain period, close database and message-broker resources, handle long-lived connections, and ensure the container receives SIGTERM. The process should remain alive while in-flight requests finish. For worker processes, readiness may mean the consumer can safely accept new work rather than HTTP traffic.
3. Set budgets, not arbitrary probe numbers
The following Deployment fragment illustrates distinct budgets. These numbers are examples: derive them from startup-time distributions, incident data and latency objectives. Kubernetes' failureThreshold and periodSeconds determine how many failed checks are tolerated, while timeoutSeconds bounds each check.
containers:
- name: api
image: example/api:stable
ports:
- containerPort: 3000
startupProbe:
httpGet:
path: /health/startup
port: 3000
periodSeconds: 5
failureThreshold: 24
readinessProbe:
httpGet:
path: /health/ready
port: 3000
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 2
livenessProbe:
httpGet:
path: /health/live
port: 3000
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
terminationGracePeriodSeconds: 45
The startup configuration permits roughly two minutes of failed checks, subject to scheduling and probe execution timing. A readiness failure stops new normal Service traffic without killing the process. A liveness failure triggers restart after the configured tolerance. An HTTP liveness check can still fail if a Node.js event loop is blocked, so use CPU profiles and event-loop delay metrics to distinguish a real hang from transient overload.
4. Readiness should fail deliberately
Readiness can incorporate a bounded check for a strict dependency when the service cannot serve any useful requests without it. But a remote dependency that is slow for every pod can make the entire Service disappear. Prefer local indicators of whether the process can safely accept work, plus circuit breakers, timeouts and partial degradation when possible. Avoid expensive database queries on every probe: kubelet runs probes continuously and may probe unready containers more frequently than the nominal interval.
For Kafka consumers, readiness and liveness may need different contracts from an HTTP API. A consumer might stay alive and retry a temporary broker outage while reporting its backlog and partition assignment through metrics. A crash loop is not a substitute for a clear retry policy and backpressure.
5. Shutdown is a separate traffic-draining problem
During pod termination, Kubernetes marks terminating endpoints as not ready for ordinary Service traffic. That change is not instantaneous across every proxy and connection. Handle SIGTERM, stop accepting new work, finish in-flight requests, and choose a grace period based on measured request duration. A preStop hook may help in specific ingress setups, but its time counts against the termination grace period. Do not add an arbitrary sleep without measuring whether it helps.
6. Test the failure modes before production
- Measure cold-start time and verify startup protects legitimate initialization.
- Make Redis or MongoDB unavailable briefly; confirm a shared dependency fault does not trigger fleet-wide liveness restarts.
- Inject a local process hang and confirm liveness eventually recovers it.
- Send sustained traffic through a rolling deployment and check for 502/503 spikes, dropped connections and interrupted work.
- Terminate a pod during a slow request and verify the configured drain budget.
- Inspect probe events with
kubectl describe pod, restarts withkubectl get pods, and EndpointSlice conditions when routing behaves unexpectedly.
Observe availability, request p99, restart rate, readiness transitions, deployment duration, CPU throttling and event-loop delay together. If every pod becomes unready at once, inspect the shared dependency and the readiness contract before increasing replicas.
Production takeaway
Startup buys time for initialization. Readiness controls traffic. Liveness is an escalation to restart. Keeping those responsibilities separate prevents Kubernetes from amplifying the outage it was supposed to contain.
Official references
- Kubernetes: liveness, readiness and startup probes
- Kubernetes: configure probes
- Kubernetes: Pod lifecycle and termination
- Node.js: HTTP server.close()