Why prompt-group mode can stop every later Gym checkpoint

NVIDIA-NeMo/RL #4265. RL links pin head ed14402; Gym links pin f4fcf8c. Found by reading the code on both sides; not run against a live Gym.

Before
RL sent Gym no ACKs at all (see page 1).
This PR
Adds ACKs, but in prompt-group mode it records them only once every sibling in the group has finished.
Status
Open — not fixed in this PR. Raised in the review comments.

Nine pages on this stack — this one: #4265: the bug in prompt-group mode.

Gym keeps each finished result until RL ACKs it, and its status() says "not ready to commit" while any result is un-ACKed. Gym's retire() refuses a finished result, so an ACK is the only way to release one. In sibling mode RL records the ACK debt the moment a sibling finishes. In prompt-group mode (all siblings must succeed, or the whole group is retried) RL keeps finished siblings in a local dict, pending_group_results, and returns early until all of them are in.

Same group, two modes

Prompt group p7, 3 siblings. g0 finishes first; then a snapshot starts. g0 finishes snapshot starts Sibling mode g0 sealed + ACK debt → ACK sent g1 g2 parked at a saved point Gym: 0 un-ACKed → ready, snapshot commits Prompt-group mode g0 done, held in pending_group_results — no ACK debt g1 g2 parked — cannot finish Gym: 1 un-ACKed → 409, RL retries … … until the prepare deadline Gym parks g1, g2 Worse: the group fails after g0 finished g2 fails → whole group abandoned and retried as try 1 → g0's result is thrown away on the RL side, but it is still COMPLETED in Gym. Nothing will ever ACK it, and retire() refuses it. Every later prepare keeps returning 409 until its deadline, so no Gym-aware snapshot can commit again in that process. running done + ACKed done, not ACKed parked by Gym prepare early return :2426 ACKs only at group seal :982 _abandon_entire_group :540 409 retry loop :1295 Gym _wait_prepared :611 only PARK_REQUESTED resumed :599 retire refuses COMPLETED :652

Nothing rejects this setup: prompt-group mode is a documented option, and no check in this PR or the two stacked with it refuses prompt-group mode together with Gym checkpointing. This PR even adds record_sealed_group_acknowledgements for it. A lost /run reply strands a finished result the same way; see the lost-reply page.

🔴 review comment — rollout_manager.py:2434 review comment — nemo_gym.py:1906

What would fix it — proposed, not in the PR

Nothing below is implemented. Record each sibling's ACK debt as soon as it arrives, like sibling mode does, and keep only the seal all-or-nothing. That is safe: a partly finished prompt group is never reused. It is thrown away whole on failure and on restart, so Gym does not need to keep g0's copy.

# rollout_manager.py, PROMPT_GROUP branch of _record_streamed_completion (proposed) pending_group_results[0] = result_g0 # {0: g0}, 1 of 3 in async with self._recovery_mutation("sibling_seals") as cut: ledger.record_unsealed_group_acknowledgement( # new ledger method cut, "p7", generation_index=0, completion_receipt=result_g0.completion_receipt) # ACK debts: {("p7_g0", try 0)} sink.notify_ready() # ACK sent → Gym un-ACKed: 1 → 0 if len(pending_group_results) < 3: return # g1, g2 still running # snapshot starts: Gym parks g1, g2 at a saved point # status(): running 0, un-ACKed 0 → ready_to_commit = True → snapshot commits # group fails later: nothing stranded — g0 was already released

Real code this changes: rollout_manager.py:2426-2442 and a new ledger method next to record_sealed_sibling_acknowledgement that builds the ACK from the receipt without requiring the sibling to be sealed. For lost replies, see the lost-reply page.

So what. With prompt-group mode and Gym checkpointing both on, any snapshot taken while a group is partly done waits out its full prepare deadline (300 s by default in #4266) and fails. One failed group after a sibling finished blocks every later Gym-aware snapshot for the rest of the process. Recording each sibling's ACK when it arrives removes both.