Who watches what when a generation engine dies

Companion illustration for NVIDIA-NeMo/RL #3590 (part 2/4 of the SingleController resiliency stack). Two snapshots: the first map and its links pin the review-time head cff673e; the second map shows the PR after the author's review fixes and pins the current head 5c9b813.

Six pages, one question each — this one: #3590 · the ledger.

The confusing part of this PR is that it looks like three watchers doing the same job. They are not. The GenerationFleetMonitor is a ledger — a plain-Python table of per-shard health that watches nothing by itself. Two things write facts into it: a probe loop that asks every serving vLLM shard "are you alive?" every 5 seconds (single_controller.py:718), and the generate path, which reports failures it hits on real requests (vllm_generation.py:910). Three things read it: the shard picker for the next request, the membership push to the NeMo-Gym router, and the watchdog. The watchdog is a different animal — it does not care about shards; it checks one number every 30 seconds (how long since any rollout finished?) and kills the run if the answer is "too long" (:715). On the NeMo-Gym path there is one more piece: a router actor that all Gym requests flow through. The ledger's serving set is pushed into it — at review time the router asked nothing and reported nothing back. The map below is the review-time picture, kept as-is; the names in it are the old ones, and the second map further down shows what the author changed.

vLLM generation shards shard 0 HEALTHY shard 1 DEAD ✕ shard 2 HEALTHY shard 3 SUSPECT green + amber still take traffic; red is quarantined _SERVING_STATES (fleet_health.py:63) SingleControllerActor — everything below shares ONE asyncio event loop fleet probe loop every 5 s: is_alive? per shard _fleet_probe_pump :718 poll the ledger GenerationFleetMonitor — state per shard + rules. Watches nothing; it is written and read. record_probe :217 report_failure :247 serving_shards :172 0 ✓ 1 ✕ 2 ✓ 3 ? membership epoch: 3 epoch moves when the serving set changes — not on SUSPECT write generate path a real request fails on a shard → report_failure + retry :910 write stall watchdog — a different question entirely every 30 s: did ANY rollout land recently? No shard knowledge needed. idle too long → RolloutStall :715 also reads ledger: publish fleet/* + abort if too few shards :703 read decisions that read the ledger pick a shard (native path) next_shard: fewest in flight :339 never picks red router push (Gym path) push serving URLs :786 → set_serving_backends :96 what leaves the process metrics: rollout/* + fleet/* (the watchdog is the only publisher) fatal: RolloutStall or GenerationFleetExhausted driver process — outside the actor _run_with_liveness_watch: ping() — is the event loop above frozen? :170 ping where requests actually flow (Gym path only) NeMo-Gym holds ONE fixed URL for the whole run; never re-resolves, never fails over — that is why the router exists router — a real HTTP proxy actor every request and response streams through it; picks the serving backend with fewest in flight :115 asks nothing, reports nothing back to the ledger request forward to a serving shard only serving set — pushed one way; a dead shard's URL simply stops arriving native GRPO path skips this lane: the picker (top right) chooses a shard and the SC calls it directly.

The same map after the review fixes

Same layout, current head 5c9b813. Green marks what changed. Everything got renamed for what it watches (GenerationFleetHealth, gen fleet probe loop, stall watchdog, controller liveness watch, GenerationRouter), and one arrow is new: the router now counts the failures it sees on real traffic and the probe tick pulls those counts into the ledger — the router is no longer silent, and a wedged engine that passes every probe is condemned anyway.

vLLM generation shards shard 0 HEALTHY shard 1 DEAD ✕ shard 2 HEALTHY shard 3 SUSPECT green + amber still take traffic; red is quarantined _SERVING_STATES (fleet_health.py:63) SingleControllerActor — everything below shares ONE asyncio event loop gen fleet probe loop every 5 s: is_alive? per shard, then drain router failures + push _gen_fleet_probe_pump :846 poll the ledger GenerationFleetHealth :122 renamed: it is a ledger, not a monitor — verbs update state record_probe :222 report_failure :252 own failure streak :79 — a passing probe cannot clear it; only a completed generation or refit can report_success :278 0 ✓ 1 ✕ 2 ✓ 3 ? write generate path (native) failure → report_failure :903 success → report_success :919 successes now clear the streak write stall watchdog — a different question entirely every 30 s: did ANY rollout land recently? No shard knowledge needed. _stall_watchdog_pump :758 also reads ledger: publish fleet metrics :797, abort if too few shards :831 read decisions that read the ledger pick a shard (native path) next_shard: fewest in flight :384 never picks red router push (Gym path) push serving URLs :925 now unconditional (no epoch gate), so a restarted router self-heals what leaves the process metrics: rollout/* + gen_fleet/* (the watchdog is the only publisher) fatal: RolloutStall or GenerationFleetExhausted driver process — outside the actor _run_with_controller_liveness_watch: ping() — is the loop above frozen? :171 ping where requests actually flow (Gym path only) NeMo-Gym holds ONE fixed URL for the whole run; never re-resolves, never fails over — that is why the router exists GenerationRouter — a real HTTP proxy actor picks the serving backend with fewest in flight :153 a failed forward: deliberate 500 to Gym :223, reflex-drop the backend :222 (armed only when fleet health manages it :218), count the failure :217 _on_backend_error :197 request forward to a serving shard only serving set — still pushed one way, every tick NEW: failure counts pulled each tick — drain :951 → report_failure :970 green = changed since review. native GRPO path skips the request lane: the picker (top right) chooses a shard and the SC calls it directly.

One dead engine, step by step

Two shards, default settings: probe every 5 s, 3 failures to condemn, floor of 1 serving shard. Kill shard 1's engine at t=0.

twhat happensshard 1 statewho acted
5 sprobe fails (1 of 3)SUSPECT — still takes trafficprobe loop
~6 sa real request hits shard 1, fails, is retried on shard 0SUSPECT (2 of 3)generate path
10 sprobe fails (3 of 3) → quarantined, epoch 0→1DEADledger rule
10 s+picker skips shard 1; next 5 s tick pushes [shard0] to the Gym routerDEADreaders
30 s tickwatchdog publishes fleet/shards/dead=1; 1 serving ≥ floor → run continuesDEADwatchdog

If shard 0 also died, the watchdog's floor check would end the run with GenerationFleetExhausted — the failure names the cause. Without this PR, the only backstop is the stall watchdog noticing silence minutes later and saying RolloutStall — true, but it names the symptom. A restarted engine does not come back on its own: answering probes again proves nothing about its weights, so re-entry requires a completed weight sync (that machinery lands in parts 3–4 of the stack).

Does it work?

claimevidence
faster, better-named failuresame kill-one-shard test on a 2-GPU box: part 1 dies in 222 s with RolloutStall; this PR dies in 62 s with GenerationFleetExhausted (from the PR description)
the router does not corrupt payloadsend-to-end Gym run through the router: gen_kl_error (vLLM logprobs vs trainer recompute) unchanged
ledger + router behaviorunit tests run against real HTTP servers, now including the wedge cases (500-never-504, reflex drop, drained counts, probe-proof failure streak). The review's caveat is fixed: both files are collected by the L0 vLLM unit lanes. PR CI at 5c9b813: 91 checks pass, 0 fail
failover end to endgrpo_sc_gym_router_failover.sh: kill one of two shards mid-run, training finishes through the router (needs ≥3 GPUs, so it self-skips on 2-GPU CI machines)
What to remember. The ledger never acts on its own — observers write it, readers consult it, and it is now named for that (GenerationFleetHealth, not "monitor"). The watchdog stays even with fleet health on, because it answers a question no per-shard probe can: is the run as a whole still making progress? The router is no longer a silent reader of pushes: it counts the failures only real traffic can reveal, and the probe tick pulls them into the ledger — see the wedge-quarantine page for that path in detail. And enabling the router without generation_fleet_health.enabled=true now warns at setup and disarms the router's reflex drop, instead of silently staying health-blind.