Where one model call's data goes: Gym keeps a small ledger, TransferQueue keeps the tokens, RL joins them

Explainer for NVIDIA-NeMo/RL #4124 (paired with NVIDIA-NeMo/Gym #3368). RL links pin c617a2c; Gym links pin b27e0db.

Before
Ordinary token capture: Gym picks each call's parent itself by matching the request against earlier calls. A rewritten history has no match, and the rollout is masked.
This PR
With context compaction (CC), the agent still decides when to compact, but it sends no flag. Gym proposes a parent, and RL's vLLM worker checks whether the request really extends it: continue it, or start a new root. Everything else on this page already existed.
Status
Proposed — open PR, not merged. The values below come from a real run of the PR's code, with a stand-in model.

Nine pages on this PR, one question each. This one: what moves where during one model call, and after the rollout ends.

Background. Five places take part. The agent (the harness in Gym) owns the conversation and its compaction policy. Gym's model server sits in front of the model and keeps the ledger. RL's vLLM worker runs the model. TransferQueue stores tokens. The RL driver collects results, rebuilds training rows and trains. Dotted words have a short explanation: hover or tap them.

The figure follows call 2 (c2) of one rollout, the same run as page 3. The agent sends only messages and never tokens (it no longer sends a parent hint either). Gym proposes the earlier call whose history the new request extends (_admit_candidate). RL's worker checks that claim and decides (decide_capture_input). The worker then writes the call's new tokens straight to TransferQueue and returns only small numbers to Gym's ledger. After the rollout, the RL driver pulls the ledger and rebuilds rows from TransferQueue. The two orange steps are new in this PR.

One call, then the end of the rollout

Agent (Gym)owns the history Gym model serverkeeps the ledger RL vLLM workerruns the model TransferQueuestores tokens RL driverrows + training ① send this turn's messages: the whole history after compaction (no tokens, no parent hint) input=[user "task", reasoning 1, assistant "answer", user "observation 1"] ② propose a parent: the earlier call whose history this request extends candidate parent=c1 · call intent for c2 written Gym sink.py L352-388 ③ forward request + candidate mode=candidate parent=c1 prev_len=3 + item hashes ④ decide: c1's history matches and only a user item is new → continue c1 prefix matches, suffix=[user] → token_in parent=c1 capture_context.py L104-114 ⑤ fetch c1's exact tokens [10,11] task + [1001] c1's answer vllm_worker_async.py L1134-1142 ⑥ append observation 1's tokens [20, 21], then generate [10,11,1001] + [20,21] → 1002 (c2's answer) ⑦ store only the new tokens group_g0/c2 → [20,21 | 1002] mask [0,0,1] capture.py L156-246 ⑧ return small numbers, no tokens parent=c1 prev_len=3 delta_len=3 cum_len=6 ⑨ write the ledger row for c2 CallRecord c2 · token_in · 3/3/6 · chatcmpl-2 external_capture.py L292-327 ⑩ ordinary response chatcmpl-2: reasoning 2, assistant "answer" Calls 3–5 repeat ①–⑩. At call 3 the policy rewrites the history, so in step ④ the history no longer matches and RL starts a new root. The agent rejects call 4 and asks again (call 5). Then the episode ends. ⑪ ordinary output + verifier reward (through Gym's /run) 8 output items (call 4's output is not among them) · reward 1.0 ⑫ RL pulls the ledger 5 CallRecords · 0 pending · 0 failures nemo_gym.py L992-1030 ⑬ fetch tokens by key 5 stored pieces ⑭ pick accepted calls, cut at roots, build rows rows g0_s0, g0_s1 · advantage +0.5 each L615-688 new in this PR work inside one part Token key (the PR test's stand-in for vLLM): [10, 11] = the task · [20, 21] = observation 1, the tool result after c1 ·1001 and 1002 = the tokens the model generated in c1 and c2.

What changed for each kind of request

Ordinary capture (CC off)Context compaction (this PR)
Who picks the parentGym, by matching the request's earlier assistant messages against recorded calls (sink.py L246-319)Gym proposes, RL's worker decides (steps ② and ④)
Plain next turncontinues the last callcontinues the last call
History rewrittenno parent found → unresolved_parent → whole rollout maskednew root, stored with its whole prompt → one more training row
Agent rejects a response and asks againthe retry is a sibling of the rejected callthe same: Gym proposes the last accepted call, so the retry is a sibling, not a new root
What the ledger holdsNever tokens: ids, parent, three lengths, hashes, response id. Tokens live only in TransferQueue, keyed <rollout>/<call>.
So what. Your picture of "Gym's ledger goes back to RL, and the reassembler builds the row" is right, with one correction: the ledger never carries tokens. The worker writes them straight to TransferQueue, and RL joins the two by key. What CC adds is steps ② and ④. Gym proposes a parent, and RL accepts it only when the request truly extends that call. Every rejection becomes a new root. Page 3 shows how those roots turn into rows.