The vLLM prefix cache kept old weights across refits — when this PR clears it, and when it only warns

NVIDIA-NeMo/RL #4018 (legacy async GRPO/PPO). Related: #3929 fixes the same flag on the single-controller path (its page). Permalinks pin the PR head 9dd13b3.

Before
When generation was drained before a refit and recompute_kv_cache_after_weight_updates was false, the vLLM prefix cache was never reset. A shared prompt prefix kept the KV from whatever weights first filled it, with no limit on how old.
This PR
Drained refits now always reset the cache. In-flight refits with the flag false still keep it, and print a one-time warning. It also flips the class default to true but pins false in the example configs, which #3929 now owns.
Status
Fixed by this PR for drained refits. Open for in-flight refits with the flag false: warned, not fixed. The default is now owned by #3929, which makes it true. The review asks this PR to rebase on #3929, drop its own default flip and false pins, and give the warning to the single-controller path too.
Background. vLLM keeps the KV blocks of finished requests, keyed by their tokens. A new request that starts with the same tokens (a system prompt, the same prompt sampled again) reuses those blocks instead of computing them. That reuse is the prefix cache. A refit changes the weights, but a cached block still holds KV computed by the weights that were loaded when it was filled. Dotted words have a short explanation: hover or tap them.

Before each refit the collector does one of two things in prepare_for_refit. With in-flight updates on, it pauses vLLM with running requests kept (L1122); the flag decides whether that pause also clears the cache. Otherwise it waits until every request has finished (L1142), a drained refit. On main, a drained refit reset the cache only when the flag was true (L1131 at base), and the example configs set it to false.

One shared prefix P, five weight versions

Which weights computed the KV for prefix P, at each step step 1 · v1 step 2 · v2 step 3 · v3 step 4 · v4 step 5 · v5 drained refit before this PR fill P · v1fresh reuse P · v11 version old reuse P · v12 versions old reuse P · v13 versions old reuse P · v14 versions old skip reset (base L1131) drained refit with this PR fill P · v1fresh fill P · v2fresh fill P · v3fresh fill P · v4fresh fill P · v5fresh reset after refit (L1175) in-flight refit flag false, this PR fill P · v1fresh reuse P · v11 version old reuse P · v12 versions old reuse P · v13 versions old reuse P · v14 versions old warn once (L1120) ⚠ printed once, at the first refit KV computed by the weights now loaded KV from older weights; the trainer scores with the current ones one-time warning

The trainer scores every sample with the current weights, so in red cells the sampled tokens follow from KV that the trainer never sees. The gap grows each step and resets only when vLLM restarts. On hybrid models such as Nemotron-H the cache also holds Mamba states, and the reset clears those too, since one BlockPool backs every cache group in vLLM 0.29.0.

The fix

Where the fix lands: RL only, resume_after_refit in trajectory_collector.py. No vLLM change: its reset already works whenever no request is running (block_pool.py), and after a drain none is.

@@ trajectory_collector.py L1160-L1176 (head) · resume_after_refit
         if self._generation_paused_for_refit:
             self._generation_pause_requested_for_refit = False
             self._generation_paused_for_refit = False
             self._refit_pause_cleared.set()
             return

-        # Invalidate&recompute vLLM caches after the weight updates (in-flight or not) if
-        # recompute_kv_cache_after_weight_updates is True (AREAL-style implementation).
-        # Otherwise, keep using the stale KV caches (Magistral-style implementation).
-        if self.async_config.recompute_kv_cache_after_weight_updates:
+        # No in-flight pause was requested for this refit, so generation was drained
+        # first and nothing is using the KV cache: invalidating it is free. Always do
+        # it. (A backend that lacks the pause hook still goes through the pause
+        # request above and keeps the legacy behaviour below.) ...
+        drained = not self._generation_pause_requested_for_refit
+        if drained or self.async_config.recompute_kv_cache_after_weight_updates:
             try:
                 ...
                 invalidated = self.policy_generation.invalidate_kv_cache()

_generation_pause_requested_for_refit is set only when prepare_for_refit asked vLLM for an in-flight pause (L1122). Every other refit was drained, so the reset is free and always runs. The flag now only decides what happens to requests that are still running during an in-flight refit.

What stays open. In-flight refits with the flag false still reuse P from v1 (row 3); the PR only warns. vLLM cannot clear just the prefix cache while requests are running, so the real fix is either the flag true or a cache key per weight version (vLLM's per-request cache_salt), suggested as a follow-up. After #3929 lands, the single-controller path has the same row 3 and no warning, which is why the review asks to move the warning into the vLLM worker. The default is true after #3929, which sets it in the config classes and the example configs, so this PR drops its own flip and false pins on rebase.

🟡 review comment — grpo.py:301 (rebase on #3929, drop the flip) review comment — trajectory_collector.py:1121 (move the warning) review comment — single-controller.md:452 (#3929 conflict)