Where #4583's 1,626 added lines go, and the ~300 that can go without changing behavior

NVIDIA-NeMo/RL #4583, part 7 of 11 in the partial-checkpointing stack. All permalinks pin the PR head 4253178.

Before
Round 1 asked for digests instead of stored pixels. The author did that, and all 9 round-1 comments are fixed.
This PR
One commit on top of its base (rebased 11-12). About 350 changed lines of source and docs, and about 1,480 changed lines of tests (one new 350-line functional script). Counts below are lines added plus removed.
Status
Proposed, not in the PR. The cuts below are review asks (C10โ€“C18). None changes what the code accepts or rejects. Nothing here has been run.

Two pages on this PR at its current head, one question each. This one: how much is needed.

The source change is small. Most of the size is tests, and much of that repeats helpers and settings the same files or a sibling script already have. In the source, two kinds of extra code stand out: checks written one field at a time where one equality check does the same job (verify_restored_media), and shape checks on saved image data that run before the prefix digest check, which already covers that data (recovery.py:817-832).

Lines added per file, and what each review comment removes

kept removed by a comment (about) added for a test gap scale: 100 px = 100 lines source captured_media.py 77โˆ’25 C10 prefix_recovery/ (3 files) 87โˆ’25 C11 vllm_worker_async.py 169โˆ’53 C11 C12 docs + setup.py 17 tests test_vllm_token_capture_hosting 395โˆ’68+20 omni prefix-recovery .sh (new) 350โˆ’37 C17 test_captured_media.py 326โˆ’72+25 test_setup.py 65โˆ’25 C18 other test + CI files (7) 345 hosting test: โˆ’60 C16 (copied helpers and tests), โˆ’8 C11 (media-row case), +20 C15 (resume-then-cut case) captured_media test: โˆ’25 C10, โˆ’30 C11 (corrupt-row cases), โˆ’17 C12, +25 C14 (handler branch case) C17 moves ~35 settings into a test-only YAML; the sibling Omni script drops the same lines. Total: about โˆ’110 source, โˆ’200 tests, +45 for the two test gaps. "Other" = parity 94 + 86, snapshot 70, phases 58, text .sh 33, two small 4.

The biggest source cut: read the image data after the check that covers it (C11)

Where the fix lands: RL only. resolve_restore, _rebuild_prefix, the restore_media hook (deleted), and the vLLM worker's _restore_generation_cut_media. No Gym change: Gym's prefix digest already includes the row's extras (compute_staging_digest).

@@ recovery.py L817-L832 ยท PrefixRecovery.resolve_restore
         try:
-            # One fetch serves both the media and the prefix.
-            chunks = self._fetch_chunks(continuation)
-            media = self._parts.restore_media(chunks)
-            snapshot = self._rebuild_prefix(admission, prefix_token_ids, chunks)
+            snapshot = self._rebuild_prefix(admission, prefix_token_ids)
             if snapshot.digest != continuation.prefix_digest:
                 raise RuntimeError("rebuilt prefix digest does not match the admission")
             ...
-            return admission, _ResolvedGenerationCut(snapshot=snapshot, media=media)
+            # As on main: the worker reads the media from the verified snapshot.
+            return admission, snapshot
@@ vllm_worker_async.py L1253-L1272 ยท _restore_generation_cut_media
-    def _restore_generation_cut_media(
-        self, chunks: Sequence[Any]
-    ) -> CapturedMedia | None:
-        if not self._capture_media:
-            return None
+    def _restore_generation_cut_media(self, snapshot: Any) -> CapturedMedia:
-        first = chunks[0]
-        items = _media_items_from_extras(first.extras, first.snapshot, missing_message=...)
-        digests = first.extras.get(MEDIA_TENSOR_DIGESTS_FIELD)
-        if not isinstance(digests, dict):
-            raise MediaCaptureRejected("Generation-prefix media digests are missing")
-        if bool(items) != bool(digests):
-            raise MediaCaptureRejected("Generation-prefix media metadata and digests disagree")
-        return CapturedMedia(items=items, tensors=None, tensor_digests=digests)
+        # The prefix digest covers these extras, so index them directly.
+        items = _media_items_from_extras(snapshot.extras, snapshot, missing_message=...)
+        return CapturedMedia(
+            items=items, tensors=None,
+            tensor_digests=snapshot.extras[MEDIA_TENSOR_DIGESTS_FIELD],
+        )
+
+    def _capture_restored_media(self, engine_prompt, *, admission, splice, snapshot):
+        """Reprocess a restored call's media and check it against the saved digests."""
+        if not self._capture_media:
+            return None
+        return self._capture_request_media(
+            engine_prompt, admission=admission, splice=splice,
+            restored=self._restore_generation_cut_media(snapshot),
+        )

Also deleted: _fetch_chunks and _rebuild_prefix's optional chunks argument, whose "fetch when omitted" branch never runs today; the fetch_for_finalization read moves inside _rebuild_prefix. _ResolvedGenerationCut and the restore_media hook go too. Six of the seven corrupt-row test cases go, because a changed row now fails the digest check first. One behavior change: a row that passes the digest check but cannot be parsed fails the call (400) instead of regenerating it; only a bug in our own writer can produce one. Keeping resolve_restore's return type, and the one _capture_restored_media method, also make this PR easy to combine with #4979 (Gym through Dynamo). The plan is for this PR to merge first; #4979's rebase then makes one call to that method in its shared start_capture, so Dynamo gets image restore with no extra code (#4979 C92).

C11 โ€” recovery.py:819

So what. The design is right and nothing here is a bug. But PR 7 of a stack past 20k lines should be close to its minimum size, and this one is about 300 lines over it. The biggest wins are cheap: one equality check (C10), one reordered call (C11), one redundant check removed (C12), and test helpers reused instead of copied (C16, C17). Before merge, the PR also needs CI on this head and one GB200 run of the new 2-GPU test.

C10 โ€” captured_media.py:176 C11 โ€” recovery.py:819 C12 โ€” vllm_worker_async.py:1942 C14 โ€” test_captured_media.py:1786 C15 โ€” hosting test:2010 C16 โ€” hosting test:1986 C17 โ€” omni .sh:165 C18 โ€” test_setup.py:1489