Reading the rollout-recovery counters after a restart

Companion notes for NVIDIA-NeMo/RL #3925 (telemetry for rollout checkpointing), stacked on #3924. All permalinks pin the PR head a4c10a78.

Three pages from the #3925 review — this one: what the recovery counters count.

This page uses proposed names, not the ones in the PR. The counters are correct today; the names are what make them hard to read. The mapping is in the first section, and everything after it uses the proposed names so the page reads as one consistent story.

One prompt produces a group of rollouts. The group size is num_generations_per_prompt, stored per group as expected_generations; the rollouts inside one group are siblings. After a restart, work comes back from two places that never overlap: the replay buffer holds groups that finished, and the recovery ledger holds groups that did not. The code says so where it logs “unfinished rollout group(s) next to canonical group(s)”.

Proposed key names

in the PR todayproposedwhat it counts
groups_reusedgroups_complete_restored Groups that finished before the crash, restored whole from the replay buffer. No generation needed.
groups_consideredgroups_unfinished_found Groups in the recovery ledger: at least one sibling had not sealed. Includes groups with zero sealed siblings.
groups_redispatchedremove Always equal to the line above on any checkpoint this code writes: every unfinished group is relaunched, and the one branch that could drop a reserved group needs a sampler cursor the checkpoint never produces (third page). Better still, make that branch raise instead of discarding — then there is nothing for a second counter to count.
siblings_reusedsiblings_reused Unchanged. Siblings whose output survived and is not regenerated. Only groups that were dispatched can contribute.
siblings_redispatchedsiblings_rerun Siblings that must be generated again.

Two things the current names hide. reused means different things at the two levels — a whole group that needed no work, versus partial credit inside a group that is being redone. And groups_reused and groups_redispatched read like complements but count different populations, so their ratio is not “fraction of work recovered”.

A worked example

Group size 4. The in-flight cap (max_inflight_prompts) is 5, and the trainer is slow, so the rollout pump is blocked. This is the state the checkpoint captures.

What the checkpoint captures COMPLETE — in the replay buffer 7 groups, 28 finished rollouts UNFINISHED, dispatched — generating when the crash hit 5 groups — the in-flight cap. 14 sealed, 6 not. UNFINISHED, never dispatched — admitted, then the pump blocked 3 groups, 0 sealed — 0/4 each finished sealed, will be reused not sealed, must be generated again What restore does — every unfinished group is relaunched groups_complete_restored = 7 no generation; straight back into the buffer groups_unfinished_found = 8 5 dispatched + 3 never dispatched relaunched = 8 all of them, always — relaunch waits for capacity, never skips siblings_reused = 14 siblings_rerun = 18 18 = 6 from the dispatched five, plus 3 x 4 = 12
# the invariant that ties the sibling counts to the group counts siblings_reused + siblings_rerun == groups_unfinished_found * group_size 14 + 18 == 8 * 4 # 32 # the three never-dispatched groups contribute nothing to siblings_reused. # that is the visible signature of "admitted, but the pump never got to them".

Nothing is discarded here. Those 3 groups were admitted but never generated, and on restart they are relaunched and run all 4 siblings. The in-flight cap does not drop them — _launch waits on buffer capacity and the in-flight slot, and the redispatch loop awaits each launch in turn, so a blocked group is a slow group, not a lost one. The class docstring makes the same promise: recovery “cannot deadlock merely because the checkpoint contained more unfinished ownership records than free replay slots.”

The discard branch never runs on a correct cursor

There is a path where a group in the ledger is released instead of relaunched. During re-admission, if the replay buffer already holds groups stamped for the same target step, the surplus reservations are dropped:

dispatch_count = max(0, len(group_ids) - buffered) dispatch_group_ids = group_ids[:dispatch_count] for group_id in group_ids[dispatch_count:]: self._rollout_manager.discard_prompt_group(cut, group_id) # never re-served

single_controller.py:1149-1152 — and the same code runs during normal operation, where it repays a group borrowed by promote_ready_group. It is not new in this PR.

Only RESERVED groups can be dropped this way, and discard_prompt_group is explicit that such a group “will intentionally never be dispatched” — so it has zero sealed siblings and no generated work is lost. What is lost is the prompt: the record is simply deleted, and the dataloader cursor is already past it.

It cannot fire on a checkpoint this code writes. A round-trip test on the real code — a live borrow and repayment, the real checkpoint capture, a restore into a fresh controller — brings every prompt back and logs groups_considered == groups_redispatched. The re-admitted batch is stamped one past the saved sampler cursor, and nothing in the buffer can carry that stamp. The same test restored with the cursor rebuilt from trainer_version (the fallback for a checkpoint with no saved cursor) drops all three reserved prompts — which is what the branch guards against, and why one counter is enough. The third page draws both runs.