How a model call with an image is saved mid-generation and resumed, next to a text-only call (round 2)

NVIDIA-NeMo/RL #4583, stacked on #4507 (text prefix recovery). All permalinks pin the PR head 4253178.

Before
A checkpoint could save a half-finished text generation and resume it later. Setup refused to turn this on when the run captured images or video.
This PR
Removes that refusal. The first saved row for an image call now stores where the image sits in the prompt and a SHA-256 hash of each image tensor, not the pixels. On resume, the image is processed again and must hash the same.
Status
New feature. The review found no bug. All 9 first-round comments are fixed. This round asks for a smaller version of the same design (see the size page).

Two pages on this PR at its current head, one question each. This one: what changes when the cut call has an image. First-round pages (older design): text vs media, size and pixel copy.

Background. In a long rollout, a checkpoint can land while vLLM is still generating a model call. Instead of throwing those tokens away, the parent PR saves them as prefix rows, one per checkpoint. After a restart, Gym sends the same request again with the list of saved rows, and vLLM continues from the saved tokens. This PR makes that work when the prompt has an image or video. Dotted words have a short explanation: hover or tap them.

Both kinds of call share the same save and resume code. An image call adds work at four points: when the call starts, it hashes each image tensor once, off the event loop (vllm_worker_async.py:1219-1224); the first cut writes the image's place in the prompt and those hashes into the row (_generation_cut_record_payload); resume reads them back from the same single read text uses (_restore_generation_cut_media); and resume processes the image again and checks it (vllm_worker_async.py:1933-1953).

One call, cut twice, then resumed

text call image call (18 = image placeholder token) same in both lanes only the image call 0 ยท call starts admission prompt [10,12,11] prompt [10,18,18,11] imgs f32[3,2,3] hash once โ†’ {imgs: sha256 "9fโ€ฆ"} vllm_worker_async.py:1219 1 ยท cut #1 writes row 0 tokens [10,12,11, 41,42] extras: none tokens [10,18,18,11, 41,42] + media_spans + hashes (no pixels) recovery.py:531 2 ยท cut #2 writes row 1 tokens [43] extras: none tokens [43] extras: none 3 ยท restore one read from TQ rows 0,1 โ†’ tokens rebuild, check prefix digest rows 0,1 โ†’ tokens + row 0 extras parse spans + hashes from row 0 recovery.py:817-832 vllm_worker_async.py:1253 4 ยท check before continuing nothing more: the saved tokens are the prompt, no re-render process image again โ†’ [10,18,18,11] prompt must equal saved prompt tokens spans + hashes must equal row 0's vllm_worker_async.py:1933-1953 both lanes meet here: engine prompt = saved tokens vllm_worker_async.py:1874 5 ยท decode the rest prefill [10,12,11,41,42,43] โ†’ [44,45] prefill [10,18,18,11,41,42,43] โ†’ [44,45] 6 ยท final row for training all tokens, then prefix rows cleared all tokens + media_spans + imgs f32[3,2,3] (first pixel write)

imgs f32[3,2,3] = the image's processed tensor, named imgs: 18 made-up 32-bit floats in a 3ร—2ร—3 block, which is what the unit tests use in place of real processor output. Token ids and the tensor come from test_captured_media.py:255-258; the generated ids 41โ€“45 and the hash are illustrative. The final row's pixels are written at vllm_worker_async.py:1419-1450.

Where the two paths meet, and what still differs

They share the hard part. _rebuild_prefix joins the rows for both, and its digest now also covers row 0's extras (:772-773), so a changed span or hash fails the same check that guards the tokens. After step 4, both use the saved tokens as the engine prompt. What still differs: vLLM needs the original image again to prefill, because a hash cannot be turned back into pixels; and the final row carries the pixels, once.

So what. This round fixed the main cost of the first design: a checkpoint no longer writes pixels, only a few dozen bytes of hash per image, and the final row is the only pixel write. The shape is right. What is left is size: the restore reads the row-0 extras before the digest check that already covers them (step 3), so it carries its own set of shape checks, and the step-4 check is spread over two functions. The second page shows what can go.

C11 โ€” recovery.py:819 C12 โ€” vllm_worker_async.py:1942 C14 โ€” test_captured_media.py:1786