How a checkpoint can still lose prompts that never failed

A gap found reviewing NVIDIA-NeMo/RL #3599 · the problem that PR fixes is drawn here · all permalinks pin the head 95844b6.

PR #3599 makes an async-GRPO checkpoint save the loader's place from before the oldest unfinished prompt, so a restart re-does them instead of skipping them. It rests on one rule: every prompt below that saved place is already trained. That rule breaks when a prompt group is allowed to fail. The run hands work to two training steps at once (trajectory_collector.py:216-222), and when one of them loses a group to a tolerated failure it is refilled from prompts much further down the stream (:666). Training that step then pushes the saved place past prompts that are still being answered for the other step — and those are in no buffer, so nothing brings them back.

Where the prompts go

24 prompts, handed out 8 at a time. Two training steps are being filled at once. loader rewinds to here trained up to here ① at the save 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 step 0 lost to the failure step 1 — still being answered step 0, refilled not handed out Step 0 is refilled from 16-20 because 3-7 were lost, so its prompts sit on BOTH sides of step 1's. Training step 0 therefore reports "trained up to 21" while 8-15 have not been answered yet. ② after restarting from that checkpoint 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 8 prompts gone. They never failed — they were mid-generation, and the loader restarted past them. trained still being answered at the save lost to the tolerated failure (already documented) lost silently — this is the gap answered again after the restart not handed out yet With 8 prompts per step this loses 8. The recipe that matches this config uses 1024 per step.

The same thing in code

# two steps are filled at once — even with max_trajectory_age_steps: 1 _calculate_target_weights(w0) # range(w0, w0+1+1) → [step 0, step 1] # step 0's worker dies after 3 of 8 groups; 3 failures are tolerated, so the run goes on _release_target(0) # step 0 is free again, still needs 5 # the loop picks step 0 again (it is the earliest) and refills it from the CURRENT batch num_prompts_to_generate = min(8, 5) # prompts 16..20 → step 0 # step 0 now holds {0,1,2} + {16..20} → it completes and trains done_through = max(ordinals) + 1 # 21 ← past step 1's prompts 8..15 # the checkpoint rewinds to the newest remembered place at or before 21 base = newest place <= 21 # places are {0, 8, 16} → 16 # prompts 8..15 are below 16, so the restart never yields them again. # they are not in replay_buffer.pt either: only finished groups are stored.

→ two steps at once trajectory_collector.py:216-222 · refill sizing :666 · release on failure :1218 · frontier grpo.py:4728-4731 · place picked :561-575

Does it happen?

Not observed in a run — this is static analysis. Nobody has reproduced it on hardware, and I could not run the suite on this host. What I can show is that every ingredient is present in one shipped recipe, and that the guard which would prevent it is switched off in exactly that configuration.

Ingredientnemotron-3-ultra/mopd.yamlWhy it matters
max_generation_failures3A lost group is tolerated, so a step gets refilled from later prompts.
max_trajectory_age_steps1Two steps are still filled at once in the window after any start or restart.
in_flight_weight_updatestrueThe refit does not wait for running workers (:830-845), so one is still going at save time. This is the guard that would otherwise close the gap.
ft_save_period1A checkpoint is written every step, so the one inside that window always gets written — and crash recovery restarts from the newest one.
num_prompts_per_step1024Sets how much is lost: up to one full step of prompts.

Six more recipes under nemo_gym/nemotron-3-super/ and nemo_gym/grpo_qwen3_30ba3b_thinking_swe*.yaml carry the first three; they save less often, so they need the save to land in the window rather than getting one there every time.

What to do about it. The collector already knows which prompts are out with a running worker. Before picking a place to save, take the lowest prompt number still being answered and use min(that, trained_up_to) as the cut-off. That keeps the rule the design needs — everything below the saved place is trained, failed, or kept in the buffer — and costs nothing at run time. The alternative, waiting for workers to finish before each save, is simpler but gives back the speed that in_flight_weight_updates exists to buy.