What a crash costs a multi-turn Gym rollout, before and after turn-level recovery

Explainer for NVIDIA-NeMo/RL #4266 (stack part 3 of 3, on top of #4264 and #4265). All permalinks pin the PR head 98b3b6c; "before" links pin the base ed14402.

Before
A rollout snapshot saved the data plane, the replay index and the recovery ledger. After a crash, finished siblings were reused, but every unfinished multi-turn Gym rollout started again from turn 1.
This PR
Adds an opt-in mode (rollout_checkpointing.gym.participant_checkpointing_enabled) that pauses Gym at turn boundaries and saves its agents, model and resources into the same snapshot, so a restart continues from the last finished turn.
Status
New in this PR, off by default. Two review findings are open — see the last section.

Nine pages on this stack — this one: #4266: what a crash costs, before and after.

Single Controller already takes a rollout snapshot every snapshot_attempt_interval_s: it takes the data-plane barrier, saves the data plane plus the replay and recovery files, and publishes the directory with one rename (before: _save_rollout_checkpoint). This PR keeps that same path and adds Gym steps on both sides of it (_save_rollout_checkpoint_locked). Before the barrier, it closes rollout admission, asks Gym to park every agent at a turn boundary, and has Gym write its state into the unpublished snapshot directory. After the rename, it tells Gym to resume. If anything fails before the rename, Gym is aborted with the same checkpoint ID and the previous snapshot stays in use.

Where each path does the work — and where they become one

One rollout snapshot, step by step new Gym step (this PR) shared by both paths Before base ed14402 barrier + save data plane, ledger rename = publish commit_snapshot (base) This PR participant mode on close admission flush Gym ACKs Gym prepare + commit barrier + save + Gym TQ keys rename = publish Gym resume, reopen admission _prepare_and_commit_gym_checkpoint barrier :4664 commit_snapshot :4726 release :4736 same code: _capture_rollout_checkpoint_cut What still differs after they join • the saved key set also includes the TQ rows Gym's parked agents point to (gym_staging_keys) • the manifest records the Gym topology fingerprint and each participant's files, checked again at restore • the "nothing changed, skip this snapshot" shortcut is off, because Gym turns move on without any data-plane write

Same group, two restarts

One prompt group with 3 siblings. The agent needs 5 turns per sibling. The process dies right after a snapshot, at this point:

sibling turns done at snapshot restart BEFORE this PR restart WITH this PR A 5 of 5 (finished) reused reused B 2 of 5 starts at turn 1, redoes 2 keeps its 2 turns C 4 of 5 starts at turn 1, redoes 4 keeps its 4 turns ─────────────────────────── turns thrown away 6 0

How the restart gets there with this PR:

setup: pick the newest snapshot, check the Gym topology fingerprint matches setup.py:1880 restore TQ, then Gym restores its own participants while still paused setup.py:1976 actor: restore ledger: B and C get attempt n+1, matched to Gym's saved turns single_controller.py:1166 any sibling that used a restart-only resource starts over instead :1783 resume Gym; B and C pick up after their last finished turn :792

Trainer checkpoints get a second, safer layer. Each one first writes a gym_restart_fallback.json marker (:5313) and then tries the snapshot above. If that snapshot never lands, restore uses the marker and falls back to the "before" column for unfinished work (setup.py:1439).

What it costs, and what is still open

Why keep both paths: the old one needs nothing from Gym and never pauses rollouts. The new one needs a Gym build that supports the checkpoint protocol, token_capture.enabled: true, max_row_attempts: 1, and exactly one Gym actor. Every snapshot also closes rollout admission until Gym has parked. The PR has no timing numbers yet for that pause.

Two open review findings affect this mode. (1) If a Gym control call fails during a periodic snapshot, the error reaches the checkpoint loop as a RuntimeError. The loop only retries OSError and TimeoutError, so one HTTP 500 on commit ends the run instead of being retried (:4870). (2) With prompt_group recovery, a finished sibling is acknowledged only after its whole group finishes. Gym prepare waits for that acknowledgement, while its parked siblings can't finish. Every snapshot then times out after prepare_timeout_s, and three failures in a row end the run. Both have a one-place fix in the review.

🔴 review comment (1) — single_controller.py:1968  review comment (2) — config.py:1407