What keeps running, and what waits, while RL saves a turn snapshot

NVIDIA-NeMo/RL #4500 (Gym turn-level rollout recovery). Read from the code at head 341b9ad; Gym at 120d516. Step numbers match page 3.

Before
A rollout snapshot held TQ writes only while it saved TQ. Nothing paused Gym.
This PR
With target_level: turn, a Gym phase comes first: new Gym runs wait and runs already in Gym park, then the same TQ save runs.
Status
Proposed in this PR, with open review findings. Nothing measures how long the Gym phase lasts, and trainer checkpoints wait for it. review comment — single_controller.py:4727

Four pages on this PR. This one: which parts of training keep running during a snapshot.

Background. A snapshot runs in its own task, _rollout_checkpoint_pump, not in the train pump or the rollout pump. (The train pump does save TQ and the ledger itself, inside every trainer checkpoint, but with no Gym phase. With turn, that save does not count as a rollout snapshot; the train pump wakes this task to take one tied to the new checkpoint.) Nothing tells the other pumps to stop. They wait only when they reach something the snapshot holds. The snapshot holds three things: the dispatch gate, Gym itself (parked), and the TQ barrier. Dotted words have a short explanation: hover or tap them.

Ten parts, three phases

Gym phase · steps 1–12 new in this PR · park ≤ 300 s TQ save · steps 13–16 same as on main · no time limit Disk write · steps 17–19 Gym resumed, gate open Rollout pump starts new rollouts keeps picking prompts, but every new /run waits at the gate same, and its ledger writes wait for the TQ save sends again Gym runs in flight agent + tool calls parked at a turn boundary (running tools finish first) still parked running again Generation engines vLLM awake, but no new requests: Gym's model server is closed awake and idle busy again Rollout reassembler finalizer actors runs: finishes groups whose rows already came back waits (a TQ write); the save first waits for ones in flight runs Trainer forward / backward / step trains on buffered batches until they run out math keeps going; claiming a batch and step cleanup wait runs Advantage actor pool runs its writeback waits (a TQ write) runs Refit weight sync to vLLM runs after each step late: comes after the step's cleanup, which waits runs Trainer checkpoint model + optimizer save waits: same lock as the snapshot waits waits until the rename Stall watchdog stall_timeout_s 600 s keeps counting; any finished train step resets it keeps counting normal Rollout time limits RL's and Gym's paused paused counting again running normally running, but slowed or idle waiting timer paused

Each row name links to the code that decides that row. On main, the middle column already happened at every rollout snapshot; the left column is what turn adds.

The three places the trainer actually waits

It goes the other way too: the snapshot gives way to the trainer. It is skipped while an optimizer step is being committed, and thrown away if a train step finishes during the Gym phase. review comment — single_controller.py:4906

So what. A turn snapshot stops rollout production, not training. Training slows only when it runs out of buffered data, hits a TQ write during the TQ save, or tries to save a trainer checkpoint. The timers that watch single rollouts are paused. The stall watchdog is not, but any finished train step resets it, and by default it only warns. review comment — config.py:719