The KV-cache reset flag did nothing on SingleController โ€” this PR makes it work

NVIDIA-NeMo/RL #3929 ยท all links pinned at c877a8e2 ยท the same feature on the legacy path is #3839

Two pages on this PR, one question each โ€” this one: a bug the PR fixes.

Before
On SingleController + vLLM, setting recompute_kv_cache_after_weight_updates=true changed nothing. The code ran, cleared no cache, and reported success.
This PR
Freezes in-flight requests for the weight swap and hands the flag to vLLM's own pause, which really does drop their cached attention state.
Status
Fixed by this PR. Both cache modes now behave as documented. Open problems on the failure path are page 2.

Two ways to handle the cache, and both are legitimate

In async RL the model trains and generates at the same time. Every training step copies fresh weights into the generation engine โ€” that copy is a refit. A request that is only part-way through generating carries a KV cache: the saved attention state for every token so far, computed with whatever weights were loaded at the time. After a refit that saved state is out of date, and there are two accepted answers to what you do about it. NeMo-RL names them both in its own code.

ModeSettingWhat it does
Magistralrecompute_kv_cache_after_weight_updates: false Keep the stale cache. Cheaper; the request finishes using attention state from the old weights.
AREALrecompute_kv_cache_after_weight_updates: true Throw the cache away so it is rebuilt with the new weights. Costs a recompute; more correct.

the two names come from the repo โ€” trajectory_collector.py:1121-1122

Both should work. The bug was that on SingleController you got Magistral behavior no matter which one you picked.

What you actually got when you asked for AREAL

One request's KV cache, token by token 4-token prompt, 3 tokens decoded, then a refit, then 2 more. Every box is one cached token. 1 2 3 4 5 6 7 8 9 refit: weights v0 โ†’ v1 prompt โ€” 4 tokens decoded with v0 โ€” 3 tokens decoded with v1 โ€” 2 Tokens 8 and 9 come from the new weights but read all 7 boxes on the left, every one of which was built by the old weights. This is correct behavior in Magistral mode. The bug: you got it in AREAL mode too. prompt built with the old weights (v0) built with the new weights (v1)

Asking for AREAL called vLLM's reset_prefix_cache() with no arguments, which defaults to reset_running_requests=False. Upstream only clears the cache in that case "when there is no running requests taking KV cache" โ€” and in async RL something is always running, so it did nothing and returned False. That False was then lost twice: the NeMo-RL worker method has no return statement, so every result was None, and the caller's all(...) ran over an empty sequence and reported True.

the no-argument call ยท the all(...) over nothing ยท upstream vLLM says when it clears

What the PR changes

Same request, same 9 tokens. The flag is now handed to the engine's own pause:

clear_cache = self._async_cfg.recompute_kv_cache_after_weight_updates # True for AREAL await asyncio.to_thread(self._gen.pause_generation_for_refit, clear_cache=clear_cache) # ... the weight swap happens here, with generation frozen ... await asyncio.to_thread(self._gen.resume_generation_after_refit)

single_controller.py:4587-4642

await self.llm.pause_generation(mode="keep", clear_cache=clear_cache) # mode="keep" -> the request is frozen, not cancelled: it continues later # clear_cache -> True reaches reset_prefix_cache(reset_running_requests=True), # which preempts the request and drops all 7 cached tokens. # False keeps them: that is Magistral, and it still works.

vllm_worker_async.py:2026 ยท upstream _reset_caches

ModeBefore this PRAfter
Magistral (keep cache) Worked โ€” though the request kept decoding during the weight swap Works; the request is frozen for the swap, then continues with its cache
AREAL (reset cache) Silently did nothing, and reported success Really clears the cache, so tokens 8 and 9 read state rebuilt with v1
What to do about older results. If you ran SingleController + vLLM with recompute_kv_cache_after_weight_updates=true before this change, you were not getting AREAL โ€” you were getting Magistral with a success message. Any comparison that assumed a fresh cache after a refit was really measuring stale-cache behavior, so it is worth re-checking before putting it next to a run built on this PR. The freeze also costs wall-clock: in AREAL mode a request now waits for the refit and then recomputes the tokens it had already done.