Why one lost /run reply stops every later Gym checkpoint
NVIDIA-NeMo/RL #4265.
RL links pin head ed14402; Gym links pin f4fcf8c. The Gym side was run with
Gym's real checkpoint code; the RL triggers are traced from source.
Before
RL had no ACK path at all (see the "what it adds" page).
This PR
ACKs every finished run whose reply RL receives. A finished run whose reply never reaches RL is never ACKed.
Status
Open — not fixed in this PR. Raised in the review comments.
Nine pages on this stack — this one: #4265: a finished run whose reply never reaches RL.
The Gym pieces this page uses
RL plays out each rollout with one blocking /run call to a Gym agent. The reply carries the
reward, so RL needs it. Each call has a rollout ID and a try number.
When a run finishes, Gym keeps the reply and marks the run
finished, not ACKed until RL sends an ACK.
Gym's retire() can cancel a try RL no longer wants, but it
refuses a finished one. Only an ACK releases it.
Dotted words have a short explanation: hover or tap them.
So every finished run must be ACKed, or no checkpoint can ever commit again. RL only ACKs runs whose
reply it received and sealed. If a reply is lost after Gym has finished the run, RL gives up on that try,
starts the next try, and never ACKs the old one. Gym keeps it forever.
One lost reply, step by step
How a reply gets lost. None of these needs a bug; each is normal behavior:
RL's own rollout_s deadline fires (rollout_manager.py:1331) while the agent is still working. Gym keeps running the try and finishes it later.
RL stops reading the batch early, for example when another row fails first.
The new per-row receipt fetch (nemo_gym.py:1906) times out. It runs after /run has returned, outside the loop that keeps draining the batch.
Real output from Gym's own checkpoint code (the Gym participant, driven directly):
The review is still pending, so this link opens for others only after it is published.
What would fix it — proposed, not in the PR
Nothing below is implemented. Keep the ACK, and never leave a try behind. When RL gives up on a try,
it first calls Gym's retire(), and the answer says what to do. Every piece already exists in the
pinned Gym, so no Gym change is needed:
# RL, when it gives up on try 0 of r7 (proposed)
answer = retire("r7", 0)
if answer["retired"]: # try 0 was still running# Gym cancelled it; nothing is left behind
start_next_try("r7", 1) # the rerun is policy: the run never finished
elif answer["completed_unacknowledged"]: # try 0 had finished
reply = post_run("r7", 0) # stored reply, reward 0.5; the agent does not run
seal(reply); ack(reply) # un-ACKed in Gym: 1 → 0; no try 1 at all# at prepare: for each run in the 409's completed_unacknowledged_attempts,# adopt it if it is still RL's current try, otherwise fetch its receipt and ACK it
Real output from the same Gym code with this fix:
prepare: ready_to_commit=False running=0 parked=0 completed_unacknowledged=1 | listed: [('rD', 0)]
re-POST /run (rD, 0) during prepare -> replayed: {'id': 'r-rD', 'reward': 0.5}
ACK -> {'acknowledged': True, 'idempotent': False}
prepare after adopt+ACK: ready_to_commit=True running=0 parked=0 completed_unacknowledged=0
agent runs: {('rD', 0): 1} # the agent ran once: nothing was rerun
retire(rT, 0) -> {'retired': True, 'tombstoned': True} # a still-running try is cancelled
So what. Today, one reply lost after Gym finished the run is enough to block every later
Gym-aware checkpoint for the rest of the process, and the finished work is rerun under a new try anyway.
With "retire, then adopt or cancel", the finished run is used instead of rerun, a still-running one is
cancelled, and prepare never waits on a run RL has forgotten.