Who calls what when a Gym-aware rollout snapshot is saved, and when it is restored

The full stack: #4264 (ad3c9cb, the actor calls) → #4265 → #4266 (98b3b6c, the Single Controller). Gym links pin f4fcf8c.

Before
A rollout snapshot held RL's own state and TQ. Gym was told nothing, so an unfinished multi-turn rollout started over after a crash.
This stack
Each snapshot also pauses Gym at turn boundaries and has every Gym server save its part into the same folder; restore loads it back and continues each rollout from its last finished turn.
Status
All three PRs open, under review. Off by default.

Nine pages on this stack — this one: who calls what when a snapshot is saved and restored.

Three layers. The Single Controller decides when to save and owns the snapshot folder. It never talks to Gym directly: it calls the NemoGym actor, which calls each Gym server over HTTP. The Gym servers are the policy model server (passes model calls to vLLM), the agent server (runs the multi-turn loop), and the resource server (tool and environment state). Blue arrows are Ray calls inside RL, orange are HTTP calls to Gym, green are files written or read. Dotted words have a short explanation: hover or tap them.

Saving

Rollouts are paused from step 1 to step 5: new Gym rollouts are not sent, and Gym refuses new model calls. RL's own cut (step 3) runs under the data-plane barrier, after Gym has finished writing, and the folder is published with one rename only when every part is on disk. Code: _prepare_and_commit_gym_checkpoint, _save_rollout_checkpoint_locked, and the actor side in prepare_checkpoint / commit_checkpoint.

Saving a snapshot (every snapshot_attempt_interval_s)NeMo-RLGym server (HTTP)filesSingleControllerNemoGymactorpolicy modelserveragentserverresourceserverTransferQueue(TQ)snapshotfolder1 · stop new work and park at a turn boundary (rollouts are paused from here until step 5)stop sending new Gym rolloutsflush ACKs of finished results (#4265)POST acknowledge, one per finished resultprepare_checkpoint(id, deadline)pause: refuse new model callsprepare: park each rollout at a turn endprepare: freeze sessionsstatus: wait until 0 calls in flight2 · Gym writes its own state into the unpublished snapshot foldercommit_checkpoint(id, deadline, tmp folder)commitagent state + continuations.jsonlcommit (with the agents' continuation indexes)model ledger for those turnscommitsession state3 · RL's own cut, under the data-plane barriersave_checkpointtraining rows + token-capture rowsloader place, replay index, recovery ledger, manifest4 · publish: fsync, then rename the tmp folder into placerename tmp → snapshot_N5 · resume, in the order that opens what agents call firstresume_checkpoint(id)resumeresume: accept model callsresume: parked turns continuesend Gym rollouts again (only after Gym confirms)If anything fails before step 4, the actor calls the same resume routes (abort_checkpoint) and the snapshot is never published.

Restoring

Setup on the driver loads TQ first, then has Gym load its state while everything stays paused. Only after the Single Controller has loaded its own bookkeeping and given each unfinished rollout a new run n+1 does it resume Gym. A rollout that used a restart-only resource is thrown away and starts over. Code: setup.py:1872-1990, single_controller.py:770-797.

Restoring after a crashNeMo-RLGym server (HTTP)filesSingleControllerNemoGymactorpolicy modelserveragentserverresourceserverTransferQueue(TQ)snapshotfolder1 · setup, on the driverdiscover_checkpoint_capabilities()GET capabilities (every Gym server)pick newest snapshot; topology fingerprint must matchload TQ filestraining rows + token-capture rowsrestore_checkpoint(folder, saved id)restore: model ledger; close the old runsmodel ledgerrestore: set up parked rollouts as run n+1agent state + continuations.jsonlrestore sessionssession stateEverything restored in step 1 stays paused.2 · the Single Controller actor startsloader place, replay index, recovery ledgerunfinished rollouts get run number n+1discard continuations that used restart-only resourcesdiscard: those rollouts start overresume_checkpoint(restore id)resume (resources, then model, then agents)start pumps: re-send unfinished rollouts as run n+1run p7_g2, run 1 → continues after call-2

What is saved, and who loads it back

In the snapshot folderWritten byLoaded by
agent state, plus the continuation index (continuations.jsonl)agent serveragent server
model ledger and its token-storage indexpolicy model serverpolicy model server
session state (only servers that can export it)resource serverresource server
training rows and token-capture rowsTQsetup, on the driver
loader place, replay index, recovery ledger (run numbers, ACKs owed)Single ControllerSingle Controller
manifest: Gym's commit result and the topology fingerprintSingle Controllersetup, to check before loading

One line of a continuation index (fields from Gym's AgentContinuationRoot; values as in Gym's own tests). It says: rollout p7_g2 was parked during its first run (run 0, Gym key p7_g2), its last finished turn ended with model call call-2, and it depends on version 3 of the tools server's state. On restore, Gym sets it up again as run 1 and it continues after call-2.

{"schema_version": 1, "rollout_id": "p7_g2", "attempt_index": 0, "capture_key": "p7_g2", "last_committed_model_call_id": "call-2", "resource_state_revisions": {"tools": 3}}
So what. Gym never decides when to checkpoint and RL never touches Gym's files directly: the Single Controller drives, the actor fans out, each server saves and loads only its own part. The cost is the pause: rollouts wait from step 1 to step 5, which includes Gym's drain (up to prepare_timeout_s, 300 s by default), Gym's commit, RL's cut and the rename. The stack has no timing numbers for that pause yet.