NVIDIA-NeMo/RL #4018
(legacy async GRPO/PPO). Related: #3929
fixes the same flag on the single-controller path
(its page).
Permalinks pin the PR head 9dd13b3.
recompute_kv_cache_after_weight_updates was false, the vLLM prefix cache was never reset. A shared prompt prefix kept the KV from whatever weights first filled it, with no limit on how old.false still keep it, and print a one-time warning. It also flips the class default to true but pins false in the example configs, which #3929 now owns.false: warned, not fixed. The default is now owned by #3929, which makes it true. The review asks this PR to rebase on #3929, drop its own default flip and false pins, and give the warning to the single-controller path too.Before each refit the collector does one of two things in
prepare_for_refit.
With in-flight updates on, it pauses vLLM with running requests kept
(L1122);
the flag decides whether that pause also clears the cache. Otherwise it waits until every request has finished
(L1142), a drained refit.
On main, a drained refit reset the cache only when the flag was true
(L1131 at base), and the example configs set it to false.
The trainer scores every sample with the current weights, so in red cells the sampled tokens
follow from KV that the trainer never sees. The gap grows each step and resets only when vLLM
restarts. On hybrid models such as Nemotron-H the cache also holds Mamba states, and the reset
clears those too, since one
BlockPool
backs every cache group in vLLM 0.29.0.
Where the fix lands: RL only,
resume_after_refit in
trajectory_collector.py. No vLLM change: its reset already works whenever no request is running
(block_pool.py), and after a drain none is.
@@ trajectory_collector.py L1160-L1176 (head) · resume_after_refit if self._generation_paused_for_refit: self._generation_pause_requested_for_refit = False self._generation_paused_for_refit = False self._refit_pause_cleared.set() return - # Invalidate&recompute vLLM caches after the weight updates (in-flight or not) if - # recompute_kv_cache_after_weight_updates is True (AREAL-style implementation). - # Otherwise, keep using the stale KV caches (Magistral-style implementation). - if self.async_config.recompute_kv_cache_after_weight_updates: + # No in-flight pause was requested for this refit, so generation was drained + # first and nothing is using the KV cache: invalidating it is free. Always do + # it. (A backend that lacks the pause hook still goes through the pause + # request above and keeps the legacy behaviour below.) ... + drained = not self._generation_pause_requested_for_refit + if drained or self.async_config.recompute_kv_cache_after_weight_updates: try: ... invalidated = self.policy_generation.invalidate_kv_cache()
_generation_pause_requested_for_refit is set only when prepare_for_refit asked vLLM for an in-flight pause
(L1122).
Every other refit was drained, so the reset is free and always runs. The flag now only decides
what happens to requests that are still running during an in-flight refit.
false still reuse
P from v1 (row 3); the PR only warns. vLLM cannot clear just the prefix cache while requests are
running, so the real fix is either the flag true or a cache key per weight version
(vLLM's per-request cache_salt), suggested as a follow-up.
After #3929 lands, the single-controller path has the same row 3 and no warning, which is why the
review asks to move the warning into the vLLM worker. The default is true after #3929, which
sets it in the config classes and the example configs, so this PR drops its own flip and false pins on rebase.
🟡 review comment — grpo.py:301 (rebase on #3929, drop the flip) review comment — trajectory_collector.py:1121 (move the warning) review comment — single-controller.md:452 (#3929 conflict)