How a wedged engine gets quarantined — proposed in review, now code in the PR

Was a review proposal on NVIDIA-NeMo/RL #3590; the author shipped it (thread, commit c02751a). Companion to who-watches-what. Permalinks pin head 5c9b813.

A wedged engine is the worst failure: the worker process is alive, so the health probe passes forever — is_alive is literally return True — but generation requests sent to it never come back. No probe can detect this. Only traffic can: a request that times out is the proof. The shipped fix has two layers on the router's error path. Layer 1, the reflex: a failed forward removes that backend from the router's local serving list at once and answers Gym a deliberate 500 (409 only when no backend is left) instead of the 504 aiohttp would pick — 500 Gym retries a bounded 3 times, 504 it retries forever. Layer 2, the report: the router counts each failure per backend URL, the 5-second control tick drains the counts and feeds them into the health ledger through report_failure.

Reflex alone (the trap the review flagged) — the cycle repeats forever 0s 5s 10s 15s 20s reflex drop (first failure returns) membership push re-adds W every 5s — the ledger still thinks W is fine ∞ wedged backend W: in serving list (eating requests) dropped (quiet) sacrificed request (fails, then retried on a healthy shard) What shipped (reflex + drained reports) — over at the first tick 0s 5s 10s 15s 20s 3 failures counted 5s tick: probe (ok, streak untouched) → drain {W: 3} → report_failure ×3 ≥ threshold → W is DEAD in the ledger quarantined — every future push excludes W; part 4's supervisor restarts it, and it re-enters only after a completed weight sync (STALE → refit) tick = _gen_fleet_probe_pump :846 drain_backend_failures :132 streak → DEAD :271 push = set_serving_backends :110

One wedge, tick by tick

Two shards: A healthy, W wedged. The table uses a 2-second failure for readability — that is the refused-connection case. A wedge that accepts and hangs costs up to backend_timeout_s (600 s default) per sacrificed request instead, and no read-idle timeout can shorten it: Gym never streams, so no bytes arrive until the generation finishes. Probe every 5 s, unhealthy_threshold: 3.

teventW's failure count in the routerW in router's list?ledger says W is
0–2 s3 requests park on W (its in-flight count was lowest)0yesHEALTHY
~2 sfirst one fails → deliberate 500 to Gym → reflex drop; Gym re-sends each failed call, they land on A, and a multi-turn rollout keeps the turns it already finished3noHEALTHY
5 stick: probe W → ok, but ok cannot clear the reported streak → drain {W: 3} → report_failure ×3 → threshold hit0noDEAD
>5 severy push sends only A. Part 4's supervisor restarts W; it serves again only after the next weight sync.—noDEAD → RESTARTING

The earlier version of this page called out an honest boundary: reported failures used to share a counter with the probes, and a passing probe reset it — so under trickle load (one sacrifice per window) W bounced HEALTHY↔SUSPECT forever. That boundary is gone. Reported failures now keep their own streak that a probe cannot touch, so even one sacrifice per window reaches 3 and condemns W — the trickle case is tested by name: a shard answering every probe while failing every generation still dies.

Where the code went past the proposal

The author took the two-layer design and hardened it in three ways the review had not asked for:

changewhy it matters
Separate failure streak, cleared only by a completed generation or a refit The proposal only sketched "feed the counts into report_failure", which back then shared the probe counter — the trickle hole above. Splitting the counter closes it.
Reflex armed only when fleet health drives membership Dropping a backend locally is safe only because a later push puts it back. Router on, fleet health off → nothing ever pushes, so a few transient blips would silently drain the fleet to nothing. Disarmed instead, and setup warns about the combination.
Status split: 500 while others serve, 409 when the list is empty 500 is in Gym's bounded retry set — it re-sends that one call and the rollout keeps its finished turns. 409 is not retried, so it is reserved for "nothing left to route to", where failing the row is the right answer.

One asymmetry remains. The reset half of the streak, report_success, is called only on the non-router path (vllm_generation.py:919). The drain carries failures only — the router never reports successes — and part 4's refit reset fires only for restarted (STALE) shards. So on a router run, a healthy shard that collects 3 transport blips over a long run — spaced days apart — is condemned as if they were consecutive. With part 4 the cost is one needless restart and refit; in this PR alone the shard is out for good.

Does it work?

claimevidence
a wedged backend is answered 500, never Gym's retry-forever 504 test_a_wedged_backend_is_answered_500_and_never_504 (in TestBackendErrorHandling: also the reflex drop, the disarm without fleet health, the last-backend 409, drain-once counting)
a shard that passes every probe but fails every generation still dies TestReportedFailureStreak, incl. the success-clears-the-streak half
end to end: kill a shard mid-run, training finishes through the router grpo_sc_gym_router_failover.sh, registered in the SingleController functional lane (needs ≥3 GPUs, so it self-skips on 2-GPU CI machines)
nothing else brokePR CI at head 5c9b813: 91 checks pass, 0 fail
What to remember. A wedge is invisible to every probe — only a failed request proves it. The shipped fix makes the first failure an immediate local drop, counts the failures where they happen, and drains them into the ledger on the next 5-second tick, where a streak no probe can reset condemns the shard. The proposal's weakest point — trickle load resetting the count — was fixed by the author going one step further than asked. The open edge is the mirror image: on the router path nothing ever clears a healthy shard's streak, so rare unrelated blips accumulate forever.