NVIDIA-NeMo/RL #4500
(Gym turn-level rollout recovery). Drawn from the code at head 341b9ad, not from the PR
description. Every step links to its line.
target_level: turn, each snapshot also pauses Gym and saves where every unfinished run is. A restart picks those runs up mid-conversation.Four pages on this PR. This one: the exact order of a save and a restore, read from the code.
r7#0 means rollout r7, Gym attempt 0. gA and gB are two copies of the Gym servers.
The recovery ledger is RL's list of which runs each copy holds.
TQ holds finished rollout data until training uses it.
Dotted words have a short explanation: hover or tap them.The PR description has its own 15-step version of this diagram. I rebuilt it from the code to check
it. The order matches. But six steps are missing or drawn differently (marked â—†),
and most of them are where a snapshot can be lost. The example: copy gA has two runs in flight.
r7 is mid-turn, and r9's last reply is already on its way back. Copy gB still runs
r4, which RL already dropped.
Where the PR's diagram differs from the code â—†
gym_checkpoint.json is written at commit, not with the TQ save. Step 15
rewrites it only when a dropped run has
to be left out.How a save fails: a prepare timeout (step 5) or a drain timeout (step 12) counts as one failed snapshot, and three in a row stop the run. A parked run that Gym did not save stops the run at once.
This matches the PR's flowchart on the main points. The saved runs must equal the ledger's runs. Each restored run moves to Gym attempt 1 and goes back to the same Gym copy. Two paths are missing from the flowchart â—†:
turn runs start over
from their prompt. This needs the gym_restart_fallback.json marker; without it, the restore fails.r7#1
when a later RL step fails, which is right. When Gym's restore itself fails, Gym gets resumed twice. #19 · gym_checkpoint_coordinator.py:537| Step | Comment | What goes wrong |
|---|---|---|
| save 1 | #4 · single_controller.py:4727 | nothing measures how long each snapshot pauses rollouts |
| save 4 | #11 · test_checkpoint_dispatch_races.py:1244 | no test notices if retired episodes are never marked retired |
| save 5 | #1 · gym_checkpoint_adapter.py:208 | a restart-only Gym agent never parks, so prepare times out; 3 timeouts stop the run |
| save 5 | #8 · config.py:719 | the stall watchdog keeps counting while Gym is parked |
| save 12 | #2 · nemo_gym.py:1182 | one failed row ends the whole stream for its group |
| save 12 | #3 · rollout_manager.py:1864 | a re-sent row waits behind the closed gate but still counts as running, so the drain times out |
| save 14 | #6 · single_controller.py:4906 | a train step that ends during the Gym phase throws the snapshot away after the full pause |
| save 17 | #7 · nemo_gym.py:1961 | if one resume is lost, Gym stays closed for the 600 s lease |
| restore 4 | #28 · single_controller.py:1308 | the mismatch error does not say which episodes differ |
| restore 5 | #19 · gym_checkpoint_coordinator.py:537 | a failed Gym restore calls resume twice (cleanup is written in two places) |
| restore 10 | #12 · turn_recovery.sh:432 | the functional test passes even if the episode started over |