How RL now keeps track of a finished Gym rollout until Gym is allowed to forget it

NVIDIA-NeMo/RL #4265 (stack part 2 of 3, on top of #4264). RL links pin head ed14402 (base: ad3c9cb); Gym links pin f4fcf8c.

Before
Each Gym try got a random ID. RL kept no record of which finished Gym results it owned, and had no way to tell Gym "I have this one, you can delete your stored copy".
This PR
Gives each rollout a stable ID plus a try number, fetches a receipt from Gym for every finished row, and stores "I still owe Gym an ACK for this receipt" in the saved recovery state.
Status
New in this PR — under review. The bug in prompt-group mode is on the second page.

Nine pages on this stack — this one: #4265: how a finished Gym rollout is tracked.

What /run is, what Gym keeps, and what an ACK deletes

  1. /run is the HTTP call RL makes to a Gym agent to play out one rollout (one sibling) from start to finish: every model call, every tool call, and the final scoring. It is a single blocking POST: it returns only when that rollout is done. RL sends all siblings' /run calls at once and handles each reply as it arrives. There is no polling.
  2. What /run returns (a BaseVerifyResponse):
    • response: the output messages and tool calls, as text
    • reward: the score the environment gave this rollout
    • mask_sample: "leave this sample out", set when the environment itself failed
    • a few IDs, and the original request fields
    This reply is the only way the reward reaches RL. RL reads it straight from the reply (nemo_gym.py:2053). So RL must hear back from every /run. Without the reply, the rollout has no reward and cannot be trained on.
    The reply has no tokens. The token IDs and logprobs were already written to TQ during the rollout, one row per model call.
  3. When the agent finishes, Gym keeps that reply in memory, filed under the rollout's ID and try number.
  4. Why: Gym's HTTP client re-sends the same /run if the connection drops. If the first reply was lost after the agent finished, the re-sent call gets the stored reply back instead of running the rollout again. This copy lives only in memory: a Gym checkpoint does not save it, so it does not survive a restore.
  5. An ACK is RL saying "I have saved this result". Gym checks the receipt matches exactly, then deletes the stored reply and marks that ID and try as done for good. It does not touch TQ or anything RL still needs.
  6. Until every finished run is ACKed, Gym will not commit a checkpoint.
Dotted words have a short explanation: hover or tap them.

Gym runs each rollout as an agent. With checkpointing on, Gym keeps every finished result until RL sends an ACK for it (a short "I have saved this result, you can delete your copy" message). Its code says so: results "remain replayable until their receipt is acknowledged", and Gym will not commit a checkpoint while any finished result is still un-ACKed. So RL must ACK each result, but only after the result is safely in RL's own saved state. Otherwise a crash between the two loses the rollout. This PR builds that hand-off.

One rollout, before and after

Example: prompt group p7, 3 siblings (g0, g1, g2); sibling g2 is on its 2nd try (try number 1) Before (#4264) row sent to Gym _ng_rollout_id = "p7_g2_a9c41…" (random) Gym agent runs, result kept state COMPLETED RL seals sibling in its ledger (no record of Gym's copy) no ACK path: prepare only retries Gym's 409 and waits base nemo_gym.py:1174 base gate_rollout_id :249 After (#4265) row sent to Gym _ng_rollout_id = "p7_g2" _ng_attempt_index = 1 Gym key: "p7_g2-a1" Gym agent runs, result kept state COMPLETED issues a receipt GET receipt RL, in one step: 1. seal sibling 2. record ACK debt both in saved state send ACK Gym deletes its copy RL removes debt (triggered by #4266) rollout_manager.py:1178 gym_capture_key :250 _completion_receipt_for :1025 rollout_manager.py:2445 record_sealed_sibling_ack… :964 acknowledge_… :1045 ledger removes :1031 green = new in this PR orange = Gym side (unchanged, pinned version) gray = same as before Why the ID change matters: after a restart RL can name the exact try it owns ("p7_g2", try 1), and a retry becomes try 2 ("p7_g2-a2"). A random ID could not be matched back to Gym's record of the same run. Gym refuses to checkpoint while any result is COMPLETED and un-ACKed (status() → ready_to_commit).

The same three siblings, step by step

Where the two paths are the same: the row still goes through run_rollouts to Gym's /run, and RL still seals it with mark_sibling_sealed. What is new sits on either side of that seal: the receipt before it, and the ACK debt after it.

before (#4264) after (#4265) ID sent for g2, try 1 "p7_g2_a9c41…" (random) "p7_g2" + attempt 1 → Gym key "p7_g2-a1" g2 retried again new random ID attempt 2 → "p7_g2-a2" g0 finishes seal g0 GET receipt, then in one locked step: seal g0 + debt {("p7_g0", 0)} saved recovery state sealed siblings only sealed siblings + ACK debts (schema v3) ACK to Gym none — no ACK path POST /acknowledge, then remove the debt Gym prepare 409 while any result un-ACKed same rule; now RL can clear it

The ACK debt is written in the same locked step as the seal (rollout_manager.py:2445-2465), and the network send happens after the lock is released. That order is required: if RL crashed after sending the ACK but before saving, Gym would have deleted its copy of a result RL never saved. Debts are removed only after Gym accepts the ACK (mark_completed_executions_acknowledged).

Two things called "receipt", and the order RL does things

Completion receiptRollout receipt
Made byGym, for one finished runRL, from Gym's token-capture list for that run
HoldsID, try number, a hash of the /run reply, which token list to read, the last model call which TQ rows make up this rollout, which call was the last one, the reward
Used forthe body of the ACK (and picking the right token list)rebuilding the training row from TQ

For each finished row, in this order:

  1. /run returns the reply (nemo_gym.py:1875).
  2. RL fetches the completion receipt from Gym (_completion_receipt_for).
  3. RL fetches the run's token list from Gym and builds the rollout receipt (_postprocess_receipt_mode).
  4. RL seals the row and writes "I owe Gym an ACK" in one locked step (rollout_manager.py:2445).
  5. After the lock is released, RL sends the ACK (acknowledge_completed_executions); Gym deletes its copy of the reply; RL removes its "owe an ACK" note (mark_completed_executions_acknowledged).

By step 5, RL already has everything it needs. The ACK only lets Gym free its copy.

So what. After this PR, a finished Gym rollout has one owner at every moment: Gym until RL has saved it, and RL after. A retry never reuses an old try number. The one path that breaks this is prompt-group mode, where RL waits for the whole group before recording any debt. See page 2.