What you get after each PR in the Gym checkpoint stack

Overview of the three-PR stack split from NVIDIA-NeMo/RL #4117: #4264 (head ad3c9cb) → #4265 (ed14402) → #4266 (98b3b6c). Each PR sits on the one before it.

Before
When a run crashes, a finished Gym rollout is reused, but every unfinished multi-turn Gym rollout starts again from turn 1.
This stack
Teaches RL to pause Gym, save Gym's state inside the rollout snapshot, and on restart continue each unfinished rollout from its last finished turn. Off by default.
Status
All three open, under review. The saving only shows up after #4266. #4264 and #4265 build the parts it needs.

Nine pages on this stack — this one: what you have after each merge.

Gym runs each multi-turn rollout as an agent that calls the policy model once per turn. To continue a rollout after a crash, three things must be true at the moment of the snapshot: every Gym server has stopped at a safe turn and written its state (#4264: prepare / commit / resume); each rollout keeps one fixed ID for its whole life, plus a run number that goes up whenever it is resumed or run again, and RL tells Gym to drop a finished result only after RL has saved it (#4265: stable IDs, receipts, ACKs); and the snapshot actually drives those steps and restores them on restart (#4266: _save_rollout_checkpoint_locked). Each PR adds one of the three. A new run number does not mean starting over: the resumed run keeps its saved turns, and the new number lets RL and Gym ignore anything the crashed run wrote after the snapshot.

What you have after each merge

works built, not used yet not there Each column includes everything to its left. main today + #4264 participants + #4265 rollout state + #4266 turn recovery RL can pause, save and resume every Gym server — built; only discoveryruns, at setup still not called ✓ used by snapshots Why: turns can only be saved when nothing is mid-turn, so every Gym server has to stop at the same point and write its own part. fixed ID + run number (token-capture runs) new random ID per run new random ID per run ✓ ID "p7_g2", run 1Gym key "p7_g2-a1" ✓ matched on restart Why: after a restart, Gym has to find the saved turns of the same rollout and ignore anything the crashed run wrote after the snapshot. finished result gets a receipt, then an ACK no receipt, no ACK no receipt, no ACK ✓ saved as "ACK owed" ✓ Why: Gym keeps a finished result until RL saves it. The ACK repeats its receipt (a hash of that result), so Gym drops only what RL saved. rollout snapshot also holds Gym's state RL state only RL state only RL state only ✓ has Gym's state Why: the snapshot is what a restart loads. Without the Gym agents, model ledger and tool sessions in it, there is nothing to continue from. restart continues from the last finished turn starts again at turn 1 starts again at turn 1 starts again at turn 1 ✓ keeps done turns Why: this is the goal. Turns that were already generated are not paid for again (example below). Crash right after a snapshot. Agent needs 5 turns per rollout. Rollout B had 2 turns done, rollout C had 4. Each square is one turn. kept done, then redone not done yet B (2 of 5) C (4 of 5) turns redone 6 6 6 0 A rollout that had already finished is reused in every column, so it is not drawn.

What each PR adds, and what is still open

PRAfter it merges you haveTurn it on withTop open review finding
#4264
page		 RL asks every Gym server what it can checkpoint, and has the calls to pause, save, restore and resume them in a fixed order. Nothing calls them yet. rollout_checkpointing.gym.capability_discovery_enabled Its Gym version bump breaks four finalize metrics on every token-capture run, flag on or off (page, comment). The new Gym commit is also not on Gym main yet (comment).
#4265
page		 Each rollout keeps one fixed ID, plus a run number that goes up when it is resumed or run again (a resumed run keeps its saved turns). Each finished result gets a Gym receipt (its ID plus a hash of the result), is saved as "ACK owed", and is ACKed to Gym only after RL has saved it. Gym releases the result only if the ACK repeats that exact receipt. no new flag; the stable IDs apply to every token-capture run In prompt-group mode the ACK waits for the whole group, so Gym never becomes ready to checkpoint (page, comment).
#4266
page		 Every rollout snapshot pauses Gym at a turn boundary, saves its state with the snapshot, and a restart continues from the last finished turn. …gym.participant_checkpointing_enabled, plus token_capture.enabled, max_row_attempts: 1, one Gym actor One failed Gym call during a snapshot ends the run instead of being retried (comment). Prompt-group mode also blocks every snapshot (comment).

The review comments linked above are still pending, so they open only for the reviewer until the reviews are published.

So what. The payoff (the 6 → 0 in the figure) arrives only with #4266. The checkpoint steps in #4264 and #4266 are off by default, but two changes reach every token-capture run whether the flags are on or not: #4264's Gym version bump (which breaks four metrics today) and #4265's stable rollout IDs. Review those two as changes to existing runs, not as opt-in features.