Why clearing only the worker's image cache after a refit crashes a vision-language run

NVIDIA-NeMo/RL #3638 — links pin head 83c655cb and vLLM v0.29.0.

Before
The default refit cleared no image cache. Users who wanted old image results dropped turned on reset_encoder_cache_after_weight_update, which clears both of vLLM's records through the engine.
This PR
Every refit, for every model, now calls the model runner's reset_encoder_cache(). That clears only the worker's record.
Status
Open — not fixed in this PR. Raised as review comment C1. Not run on a GPU; traced through the vLLM source.

Two pages on this PR — this one: the bug in the new per-refit cache reset.

Background. A vision-language model turns each image into numbers once (the encoder output) and reuses them. vLLM tracks this in two places that must agree: the scheduler's record of what is cached, and the worker's dict of the actual tensors. The scheduler decides; the worker only looks up. Dotted words have a short explanation: hover or tap them.

This PR added _reset_model_runner_caches_after_refit to both refit finalizers (native, default). It calls model_runner.reset_encoder_cache(), which only does self.encoder_cache.clear(). The scheduler's record is untouched; only EngineCore.reset_encoder_cache clears both.

Image A before and after one refit

1 · rollout uses image A 2 · refit 3 · a later request sends A again this PR non-colocated or in-flight scheduler worker cached: {A} encoder_cache: {A: tensor} cached: {A} ← untouched encoder_cache: {} "A is cached" → skip encoder Encoder cache miss for A PR: worker-only clear check_and_update_cache → True gpu_model_runner.py L3327 raises colocated sleeps first both have A sleep clears both: {}, {} A re-encoded, runs fine core.py L827 _reset_caches records disagree / crash records agree Finished requests leave A in the scheduler's list as "freeable" until space runs out, so the stale entry stays. Nothing in vLLM 0.29 tells the scheduler that the worker dropped it.

Who hits it: runs where the engine does not pause with a full cache clear before the refit, and an image comes back after it.

  1. Async, in-flight updates with recompute_kv_cache_after_weight_updates: false: a later turn of a multi-turn episode, or a sibling rollout, sends the same image. The shipped async video recipe is set up exactly this way.
  2. Non-colocated runs where an image repeats: a second epoch, several questions on one image, or a request that is preempted and starts over.
  3. Colocated runs are safe: sleeping clears both records first.

🔴 review comment C1 — vllm_backend.py:1741

What would fix it — proposed, not in the PR

Where the fix lands: RL only, two files. Nothing below is implemented. Delete the helper and its two calls. No vLLM change needed for this PR: vLLM's own reload_weights does the same worker-only clear (L5686), which is worth an upstream report, but this PR only has to stop calling it on every refit.

@@ vllm_backend.py L1727-L1741 · VllmInternalWorkerExtension._reset_model_runner_caches_after_refit
-    def _reset_model_runner_caches_after_refit(self) -> None:
-        """Mirror the cache invalidation performed by vLLM reload_weights."""
-        for method_name in (
-            "reset_lora_state",
-            "reset_encoder_cache",
-            "reset_mm_cache",
-        ):
-            ...
-            reset()
@@ vllm_backend.py L1803-L1804 · native finalize()
                     self._maybe_process_mtp_drafter_after_loading()
-                    self._reset_model_runner_caches_after_refit()
@@ vllm_backend.py L1849-L1850 · default finalize()
             self._maybe_process_mtp_drafter_after_loading()
-            self._reset_model_runner_caches_after_refit()
@@ test_vllm_backend.py L1872-L1874 · test_mxfp8_native_linear_refit_uses_vllm_layerwise_reload
         ("finalize", model, extension.model_config),
-        "reset_encoder_cache",
-        "reset_mm_cache",
     ]
  1. The test keeps its two recording lambdas, so it now fails if a worker-only reset comes back.
  2. The other two calls in the helper do nothing useful here: reset_lora_state returns at once without LoRA enabled, and native LoRA never reaches this code. reset_mm_cache clears only a profiling cache.
  3. Users who need old image results dropped keep the safe opt-in, reset_encoder_cache_after_weight_update, which clears both records and is documented as safe only when generation is paused.
So what. A pure text run never notices. A vision-language run that refits without a full pause can stop with RuntimeError: Encoder cache miss for … on the first image that comes back after a weight update. None of the GB200 tests use images, so CI does not catch it. The fix is three deletions.