NVIDIA-NeMo/RL #3929 ยท
all links pinned at c877a8e2 ยท
the same feature on the legacy path is #3839
Two pages on this PR, one question each โ this one: a bug the PR fixes.
recompute_kv_cache_after_weight_updates=true
changed nothing. The code ran, cleared no cache, and reported success.In async RL the model trains and generates at the same time. Every training step copies fresh weights into the generation engine โ that copy is a refit. A request that is only part-way through generating carries a KV cache: the saved attention state for every token so far, computed with whatever weights were loaded at the time. After a refit that saved state is out of date, and there are two accepted answers to what you do about it. NeMo-RL names them both in its own code.
| Mode | Setting | What it does |
|---|---|---|
| Magistral | recompute_kv_cache_after_weight_updates: false |
Keep the stale cache. Cheaper; the request finishes using attention state from the old weights. |
| AREAL | recompute_kv_cache_after_weight_updates: true |
Throw the cache away so it is rebuilt with the new weights. Costs a recompute; more correct. |
the two names come from the repo โ trajectory_collector.py:1121-1122
Both should work. The bug was that on SingleController you got Magistral behavior no matter which one you picked.
Asking for AREAL called vLLM's reset_prefix_cache() with no arguments, which defaults
to reset_running_requests=False. Upstream only clears the cache in that case
"when there is no running requests taking KV cache" โ and in async RL something is always
running, so it did nothing and returned False. That False was then lost
twice: the NeMo-RL worker method has no return statement, so every result was
None, and the caller's all(...) ran over an empty sequence and reported
True.
the no-argument call ยท
the all(...) over nothing ยท
upstream vLLM says when it clears
Same request, same 9 tokens. The flag is now handed to the engine's own pause:
single_controller.py:4587-4642
vllm_worker_async.py:2026 ยท
upstream _reset_caches
| Mode | Before this PR | After |
|---|---|---|
| Magistral (keep cache) | Worked โ though the request kept decoding during the weight swap | Works; the request is frozen for the swap, then continues with its cache |
| AREAL (reset cache) | Silently did nothing, and reported success | Really clears the cache, so tokens 8 and 9 read state rebuilt with v1 |
recompute_kv_cache_after_weight_updates=true before this change, you were not getting
AREAL โ you were getting Magistral with a success message. Any comparison that assumed a fresh
cache after a refit was really measuring stale-cache behavior, so it is worth re-checking before
putting it next to a run built on this PR. The freeze also costs wall-clock: in AREAL mode a
request now waits for the refit and then recomputes the tokens it had already done.