What happens when a refit goes wrong while generation is frozen

NVIDIA-NeMo/RL #3929 ยท all links pinned at c877a8e2

Two pages on this PR, one question each โ€” this one: open problems, not fixed ones.

Before
No pause existed. A refit that failed was recovered in place โ€” probe the fleet, rebuild without the lost shard, retry once โ€” and generation kept running the whole time.
This PR
Freezes every in-flight request for the weight swap and unfreezes it afterwards. That is the right fix for a real bug (page 1).
Status
Open โ€” not fixed in this PR. The failure paths were not updated to account for engines now being frozen, so three of them skip the unfreeze. These are review findings raised on the PR, not something it addresses.

This PR freezes every in-flight generation request while the new weights are copied in, then unfreezes it. Page 1 covers why that is the right thing to do. This page is about the other half: the freeze is a piece of state living on remote engine processes, and the code that sets it does not guarantee it is ever cleared. Three separate ways out of one span of code skip the unfreeze.

One span, three exits

_sync_weights: from pause to resume Green = the path that unfreezes. Red = an exit that leaves the engines frozen. pause_generation_for_refit L4589 exit 1 โ€” hangs here bare ray.get, no deadline try: L4599 refit attempt except (RefitAborted, RayActorError) L4601 probe ยท rebuild ยท retry once exit 3 โ€” retry fails exit 2 โ€” a shard that dies during the pause raises here, outside the try so the probe-and-retry recovery never runs resume_generation_after_refit L4642 reached only if nothing above raised The resume is a plain statement after the try. There is no try/finally, so all three exits skip it.

Exit 2 is the one worth pausing on. The recovery inside the try exists to survive a generation shard dying around refit time โ€” its own docstring says letting that escape once "killed a run whose data transfer had already succeeded." But the pause is the first thing that talks to the engines, and it sits above the try. A shard that dies during the pause raises there, uncaught, and the recovery built for exactly that failure never runs.

the recovery and its docstring ยท the bare ray.get in pause ยท "the last unbounded wait on the refit path"

Nothing reaps a request that stays frozen

A frozen request is not cancelled โ€” it is kept, to be continued later. If its engine is never resumed, what stops it waiting forever? The natural answer is the 900-second per-generation limit, NRL_VLLM_ASYNC_TIMEOUT_SECONDS. That limit does not apply, and the reason is the line above it:

sample_result_ref = await anext(worker_gen_proxy) # no timeout sample_result = await asyncio.wait_for(sample_result_ref, timeout=900)

vllm_generation.py:1068-1082

In Ray 2.56.1 โ€” the version this repo resolves to โ€” anext waits for the value with no limit, and fetches it before returning. Ray's own source says so, with a # TODO(swang): Avoid fetching the value sitting on the line. By the time wait_for runs, the value is already in local memory, so it returns at once. The 900s budget covers a sliver, not a frozen request.

Ray __anext__ passes no timeout ยท and awaits the value

The two limits that would catch it are both off unless an operator sets them:

LimitDefaultCovers a frozen request?
NRL_VLLM_ASYNC_TIMEOUT_SECONDS900sNo โ€” anext already waited
rollout_failure.native.generation_timeout_sNoneYes, if set
rollout_failure.nemo_gym.rollout_timeout_sNoneYes, if set

generation_timeout_s = None ยท rollout_timeout_s = None

There is a second way to reach a permanently frozen engine, and it does not need the run to fail at all. When a shard stops answering the refit, the fleet can mark it dead while its process is still alive. That drops it from the group the resume talks to, so the resume skips it โ€” and its requests stay frozen on a live engine that nothing will ever wake. This needs generation_fleet_health.enabled: true, which is off by default.

marking a live shard dead ยท the resume only talks to the current group

What would fix it โ€” proposed, not in the PR

Nothing below is implemented. Two changes would close this, and they have to go together. Bounding the pause where it stands would raise a timeout outside the try, which still ends the run; moving it inside without a bound still hangs. So: move the pause inside the try so a dying shard reaches the existing recovery, and give it the same deadline the refit already has.

try: await self._call_gen_within(self._gen.pause_generation_for_refit, clear_cache=clear_cache) # bounded, like the refit await self._sync_weights_within(kv_scales, "first") except (RefitAborted, RayActorError) as failure: ... # probe, rebuild, retry once finally: await self._resume_best_effort() # never leave engines frozen

One caveat on that finally, because the obvious version is worse than nothing. On the branch where the refit lost the trainers' CUDA contexts, the run is ending on purpose โ€” a resume that hangs there just adds a second unbounded wait on the way out. Whatever goes in the finally has to be bounded, best-effort, and must not replace the original error.

the branch that ends the run on purpose

What this means in practice. With the shipped defaults these paths end the run, so a frozen engine dies with the process and costs nothing. They start to matter when you turn on generation_fleet_health.enabled โ€” which is exactly the setting you turn on because you want a refit failure to be survivable. That is the case where a shard can be dropped while alive and left frozen, and where the pause sitting outside the try removes the recovery you enabled it for. If you run with fleet health on, also set generation_timeout_s or rollout_timeout_s, so a request that gets stranded is eventually reaped instead of waiting forever.