Subagents plus compaction: what the call forest looks like, and how to turn it into rows

Follow-up proposal for NVIDIA-NeMo/RL #4124 (paired with NVIDIA-NeMo/Gym #3368), building on page 3. RL links pin c617a2c; Gym links pin b27e0db.

Before
This PR turns one list of the leader's calls into rows. Each call in the list must continue the one before it, unless it is a root.
Proposal
Collect the calls every agent kept, then make one row per leaf and train each call in exactly one row.
Status
Proposed — not implemented. A draft for discussion. The 12-call example below is made up by hand, not from a real run.

Nine pages on this PR, one question each. This one: what changes when the agent has subagents.

How real harnesses run subagents. In Claude Code, a subagent normally starts with a fresh context: its own system prompt and a task, nothing else. A fork subagent instead starts with the leader's whole conversation. A background subagent runs while the leader keeps working, and its answer arrives later, or never, if the leader finishes first. Each subagent compacts on its own, and each keeps its own transcript. In Codex, each subagent is its own thread with its own context, and the leader collects the results. Gym's Claude Code harness already splits a run into one record per agent: each has its own list of calls and a link to the agent that started it (AgentInvocation, filled in at observability.py L642-651).

The example: a leader, three subagents, 12 calls

The leader L fixes a login bug. It starts B in the background to sign in, asks a fresh subagent A to find the bug, then forks F to write the fix. Then L compacts its history and runs the tests.

L · leader starts the task F · fork subagent starts from L's history A · fresh subagent starts empty B · background subagent nobody reads its answer rows one per leaf L compacts here same tokens + more task task answer answer summary answer never read C1 root → Agent(B, background) C2 root → curl /login C3 parent C1 → Agent(A, find the bug) C4 parent C2 → "Signed in." row 1 C5 root → grep login C6 parent C5 → "auth.py:12" row 2 C7 parent C3 → Agent(F, fork) C8 parent C7 → edit auth.py C9 parent C8 → "Fixed." row 3 C10 parent C7 → summary of C1–C9 row 4 C11 root → pytest C12 parent C11 → "Done." row 5 root: starts a new tree (new agent, or after compaction) leaf: no accepted call continues it parent: the next call resends these exact tokens and adds more hand-over: only text is passed, so no parent link C1–C12 are numbered in the order Gym's ledger records them. Four trees (roots C1, C2, C5, C11), five leaves, five rows.

Two things make this harder than page 4's forest:

  1. Some trees have more than one leaf. F's first call C8 resends L's exact tokens, so C8 is a child of C7. So is C10, the call that writes L's summary: in Claude Code, compaction is itself a model call that continues the old history (Gym files it under L's calls, observability.py L764-773). C7 has two children, so the tree from C1 has two leaves.
  2. Calls from different agents interleave. The ledger order is C1, C2, C3, …, but C2 belongs to B and C3 to L. No single list of calls can show the whole forest in order.

What this PR would do with it

This PR reads one list per rollout and cuts it at each root. Every other call must continue the call just before it, or RL raises "Selected continuation must follow its selected predecessor" (rollout_reassembler.py L1153-1162).

List the harness returnsWhat happens
Only L's calls: C1 C3 C7 C10 C11 C122 rows: C1→C3→C7→C10 and C11→C12. The other 6 calls (A, B, F) were generated by the policy but never train; RL only cleans them up (L680-687).
All 12 calls, in ledger orderRaises at C3: its parent is C1, but the call before it in the list is C2.

So you are right that one call order does not work. One order is one path through the forest, and this forest has five paths.

Proposed: one row per leaf, each call trained once

  1. Collect the kept calls. Take the union of every agent's call list. Gym already has a slot for this, one list per agent (AgentInvocation.model_calls), and Claude Code already fills it. Calls that no agent kept, such as retries, stay out, just as in this PR.
  2. Find the leaves. A leaf is a kept call that no other kept call names as its parent. Here: C4, C6, C9, C10, C12.
  3. Walk each leaf up to its root. This is the ordinary leaf-to-root walk from before this PR, run once per leaf instead of once per rollout (Gym's verify_and_linearize already takes a leaf).
  4. Train each call once. When two leaves share calls, the leaf in the root's own agent trains them. The other row keeps them as context with the loss turned off.
  5. Reward: unchanged. Every row gets the rollout's one reward, as on page 5.
row leaf walk (leaf → root, shown root-first) trains context only 1 C4 C2 C4 C2 C4 - 2 C6 C5 C6 C5 C6 - 3 C9 C1 C3 C7 C8 C9 C8 C9 C1 C3 C7 4 C10 C1 C3 C7 C10 C1 C3 C7 C10 - 5 C12 C11 C12 C11 C12 - check: 12 kept calls, each trained exactly once

Where it would land: mostly RL. RL: replace the cut in _plan_selected_calls with steps 1–4, and add the per-row loss mask for shared calls. Gym: no schema change, since the per-agent lists exist. Each harness that runs subagents must fill them, as the Claude Code harness does. B is still covered when its start is never linked to L: Gym records the gap (subagent_parent_unavailable) but keeps B's call list, and step 1 only needs the list.

This also covers the cases without subagents. Ordinary capture is one tree with one leaf, so it gives one row. Compaction in this PR gives one leaf per tree. So this is still the one shared path that the call-order follow-up asks for. The change is the input: one list of kept calls per agent, instead of one order for the whole rollout.

Open questions to settle before building it
  1. Should B's rows get the full reward? B's sign-in changed the environment, so the reward does reflect it. That matches how compaction rows are treated, but it is a choice.
  2. Should the summary call (C10) train? The policy wrote it, and a bad summary hurts every later call. This proposal trains it.
  3. Cost. Row 3 runs C1, C3 and C7 again as context. A deep fork tree repeats a lot of prompt.
  4. What if F's first call is not an exact copy? Then C8 becomes a new root. F's row then holds L's history as prompt text, which is never trained. The result is still correct and costs one more tree.
  5. Twin subagents. Two subagents with the same task and the same answer hit the duplicate-output limit from page 4.