How turn-level recovery differs from the existing sibling recovery

NVIDIA-NeMo/RL #4500 (Gym turn-level rollout recovery). RL permalinks pin head 341b9ad; Gym permalinks pin the submodule at 120d516.

Before
After a crash, an unfinished rollout always starts over from its prompt. sibling keeps the finished rollouts of a group; prompt_group redoes the whole group.
This PR
Adds rollout_recovery.target_level: turn: an unfinished rollout continues from its last finished turn, on the same Gym server copy.
Status
Proposed in this PR, with open review findings. The biggest one is on page 2: turn stalls with most Gym agents. the review

Four pages on this PR. This one: what turn adds next to the recovery that already exists.

Background. In RL training with NeMo Gym, Gym runs each rollout: an agent talks to the policy model and to a tool server (a resources server) over several turns. RL keeps a recovery ledger that says, for every rollout in flight, whether it is finished, running in Gym, or not sent yet. A rollout snapshot saves that ledger plus the token data in TransferQueue (TQ). Dotted words have a short explanation: hover or tap them.

Both levels save the same things through the same exclusive TQ barrier (single_controller.py:4902), and both restore through one function, prepare_for_restart. turn adds one thing before the barrier: it makes Gym stop every rollout at a turn boundary and write its state to disk (_prepared_gym_checkpoint). On restart, a rollout Gym saved is restored on the same Gym copy and continues as Gym attempt + 1 (rollout_recovery.py:527-540). Under sibling the same rollout is thrown away and started over (:548).

One prompt group, three rollouts, through save and restart

Group g7 at the moment a rollout snapshot starts: s0 finished · s1 in turn 2 of 3 · s2 not sent yet same in both where they split only in turn: /run sends paused, Gym rollouts parked sibling (on main) turn (this PR) save no Gym step: Gym keeps running s1 1. close the /run gate: no new sends; s1 goes on retire runs RL gave up on earlier (usually none) 2. Gym prepare parks s1 at its nearest boundary: waiting on the model → end of turn 1 (reply held) running a tool → waits for it, then end of turn 2 3. Gym commit → gym-instances/tools/replica-0/ {"tools/replica-0": [("…_g1_a0", attempt 0)]} :4727-4758 TQ barrier → capture TQ rows + ledger {s0: SEALED, s1: DISPATCHED, s2: RESERVED} turn also: check Gym's export matches the ledger, write gym_checkpoint.json, then Gym resume + reopen gate single_controller.py:4902-4920 _capture_rollout_checkpoint_cut :4466 ✕ crash, then restart from this snapshot restart no Gym step Gym restore on replica-0 (fails the run if gym_checkpoint.json ≠ ledger) :1303-1316 s0 SEALED → kept as is, its tokens come back from TQ rollout_recovery.py:519 s1 DISPATCHED → ABANDONED new attempt a1, any Gym copy every finished turn is generated again rollout_recovery.py:548 DISPATCHED → RESERVED, gym_attempt 0 → 1 same id, key "…_g1_a0-a1", sent to replica-0 Gym continues where it parked; turns kept rollout_recovery.py:527-540 owner pick :2584 s2 RESERVED, never sent → stays RESERVED and is sent normally (same rule in both) :543-547 Everything outside the dashed red boxes is shared code: one ledger, one TQ capture, one restart function, one dispatch path. The split is one branch in prepare_for_restart: a rollout Gym restored continues; any other unfinished rollout starts over. Gym copy "tools/replica-0" and the ids are shapes from the PR's own tests; the turn count is illustrative.

What turn costs

So what. turn is not a second recovery system. It is one extra Gym step in front of the save, and one extra branch in the restart, on top of the sibling path that already exists. What it buys is keeping finished turns of a long, multi-turn rollout instead of generating them again. What it costs is a pause in generation at every snapshot, and today it works only with the Gym agent and tool servers that can save a rollout mid-way.