NVIDIA-NeMo/RL #4583, part 7 of 11 in the partial-checkpointing stack.
All permalinks pin the PR head 4253178.
Two pages on this PR at its current head, one question each. This one: how much is needed.
The source change is small. Most of the size is tests, and much of that repeats helpers and settings the same files or a sibling script already have.
In the source, two kinds of extra code stand out: checks written one field at a time where one equality check does the same job
(verify_restored_media),
and shape checks on saved image data that run before the prefix digest check, which already covers that data
(recovery.py:817-832).
Where the fix lands: RL only.
resolve_restore,
_rebuild_prefix, the
restore_media hook (deleted), and the vLLM worker's
_restore_generation_cut_media.
No Gym change: Gym's prefix digest already includes the row's extras
(compute_staging_digest).
@@ recovery.py L817-L832 ยท PrefixRecovery.resolve_restore try: - # One fetch serves both the media and the prefix. - chunks = self._fetch_chunks(continuation) - media = self._parts.restore_media(chunks) - snapshot = self._rebuild_prefix(admission, prefix_token_ids, chunks) + snapshot = self._rebuild_prefix(admission, prefix_token_ids) if snapshot.digest != continuation.prefix_digest: raise RuntimeError("rebuilt prefix digest does not match the admission") ... - return admission, _ResolvedGenerationCut(snapshot=snapshot, media=media) + # As on main: the worker reads the media from the verified snapshot. + return admission, snapshot @@ vllm_worker_async.py L1253-L1272 ยท _restore_generation_cut_media - def _restore_generation_cut_media( - self, chunks: Sequence[Any] - ) -> CapturedMedia | None: - if not self._capture_media: - return None + def _restore_generation_cut_media(self, snapshot: Any) -> CapturedMedia: - first = chunks[0] - items = _media_items_from_extras(first.extras, first.snapshot, missing_message=...) - digests = first.extras.get(MEDIA_TENSOR_DIGESTS_FIELD) - if not isinstance(digests, dict): - raise MediaCaptureRejected("Generation-prefix media digests are missing") - if bool(items) != bool(digests): - raise MediaCaptureRejected("Generation-prefix media metadata and digests disagree") - return CapturedMedia(items=items, tensors=None, tensor_digests=digests) + # The prefix digest covers these extras, so index them directly. + items = _media_items_from_extras(snapshot.extras, snapshot, missing_message=...) + return CapturedMedia( + items=items, tensors=None, + tensor_digests=snapshot.extras[MEDIA_TENSOR_DIGESTS_FIELD], + ) + + def _capture_restored_media(self, engine_prompt, *, admission, splice, snapshot): + """Reprocess a restored call's media and check it against the saved digests.""" + if not self._capture_media: + return None + return self._capture_request_media( + engine_prompt, admission=admission, splice=splice, + restored=self._restore_generation_cut_media(snapshot), + )
Also deleted: _fetch_chunks and _rebuild_prefix's optional chunks argument, whose "fetch when omitted" branch never runs today; the fetch_for_finalization read moves inside _rebuild_prefix.
_ResolvedGenerationCut and the restore_media hook go too. Six of the seven corrupt-row test cases go, because a changed row now fails the digest check first. One behavior change: a row that passes the digest check but cannot be parsed fails the call (400) instead of regenerating it; only a bug in our own writer can produce one.
Keeping resolve_restore's return type, and the one _capture_restored_media method, also make this PR easy to combine with #4979 (Gym through Dynamo). The plan is for this PR to merge first; #4979's rebase then makes one call to that method in its shared start_capture, so Dynamo gets image restore with no extra code (#4979 C92).
C10 โ captured_media.py:176 C11 โ recovery.py:819 C12 โ vllm_worker_async.py:1942 C14 โ test_captured_media.py:1786 C15 โ hosting test:2010 C16 โ hosting test:1986 C17 โ omni .sh:165 C18 โ test_setup.py:1489