Which of this PR's three fixes change what a Megatron run does

Explainer for NVIDIA-NeMo/RL #4511 (fix(sft): preserve checkpoint initialization and optimizer resume state). Permalinks pin the PR head ef033f7, Megatron-Bridge ec83553 and Megatron-LM b4d72b7.

Before
SFT v2 ignored checkpointing.pretrained_checkpoint, and a conversion folder counted as complete if one small config file existed.
This PR
Forwards the setting to the model workers, checks every weight file before reusing a conversion, and adds one more weight sync after a resume.
Status
The first two are fixed by this PR. The resume sync does nothing at the pinned Megatron versions; the review asks to remove it.
Background. Megatron training keeps two copies of the weights: the BF16 weights the model runs on, and FP32 master copies the optimizer updates. On resume, NeMo-RL calls Megatron-Bridge's load_checkpoint, which reloads both. A Hugging Face model is first converted to Megatron's format once and cached in a folder; later runs reuse that folder. Dotted words have a short explanation: hover or tap them.

The PR makes three separate changes, one per row below:

  1. Use the configured start checkpoint in SFT v2. SFT v2 only. Before: wrong result. A fresh SFT v2 run with checkpointing.pretrained_checkpoint set ignored it, with no error, and trained from the Hugging Face weights instead. The model workers read the setting from the policy config (setup.py L899), and SFT v2 never copied it there. The other algorithms already do; the PR adds the same 3 lines. PPO and xtoken distillation still lack them (#4768).
  2. Stricter check before reusing a converted model. Every Megatron run. Before: wrong result, in a rare case. A cached Hugging Face→Megatron conversion counted as complete once its small run_config.yaml existed, even when a weight file was missing or empty. That can happen with an older conversion that was cut off, or files removed from storage. The run reused the folder, and the gap only showed up when the weights were read. megatron_conversion_is_complete now checks every weight file the checkpoint lists, so such a folder is converted again. New conversions were already safe, because they are published with one atomic rename. The check runs on every rank at startup; the review asks it to check each file once, not once per tensor.
  3. Extra weight sync after a resume. Every Megatron resume that loads optimizer state. Before: nothing was wrong. Bridge's load already leaves the weights matching the FP32 master copies. The PR calls quantize_and_sync_model_params_from_main_params again anyway. The weights come out the same, so the only effect is a slightly slower resume: one extra blocking all-gather across data-parallel ranks. The review asks to remove it.

Each change, on one small example

before this PR after this PR 1 · SFT v2 start point checkpointing.pretrained_checkpoint = {path: "/ckpt/sft-base"} policy.pretrained_checkpoint: (missing) → workers start from the Hugging Face weights policy.pretrained_checkpoint: {path: "/ckpt/sft-base"} → workers start from /ckpt/sft-base sft_v2.py L459-461 2 · reuse a cached conversion? iter_0000000/ run_config.yaml ✓ .metadata lists 2 files: __0_0.distcp ✓ __1_0.distcp ✗ missing checks run_config.yaml only → "complete" → reused; the gap shows up only when read checks every listed file → "incomplete" → converted again base: community_import.py L72-76 head: community_import.py L73-91 3 · resume with optimizer state (BF16 run) saved: master 0.30000001 → BF16 weight 0.30078 Bridge load_checkpoint: weight 0.30078, master 0.30000001 (already match) + new sync: writes bf16(master) = 0.30078 → same value, plus one blocking all-gather Bridge already syncs FP8/FP4 here: L3303-3310 setup.py L2714-2716 wrong result fixed by this PR no change in what the run computes

Row 3 has no FP8 case because Bridge's own load already runs the same sync when FP8/FP4 param gather is on (NeMo-RL turns it on with FP8 params), so the PR's call repeats it with the same result. And because every Megatron optimizer has the method, _sync_model_params_from_loaded_optimizer never walks into child optimizers.

🟠 review comment — setup.py:2714

The fix — what lands, and what the review asks to remove

Where it lands: RL only. No Megatron-Bridge or Megatron-LM change: Bridge already does the resume sync. First hunk is in the PR. Second hunk is proposed in the review, not in the PR.

@@ sft_v2.py L456-L461 · setup_sft_v2  (in the PR)
     """Build the V2 cluster, TQPolicy, placement, and resume state."""
     set_seed(master_config.sft.seed)
+    pretrained_checkpoint = master_config.checkpointing.get("pretrained_checkpoint")
+    if pretrained_checkpoint is not None:
+        master_config.policy["pretrained_checkpoint"] = pretrained_checkpoint
     if master_config.data.get("backend") != "energon":

@@ setup.py L2714-L2716 · setup_model_and_optimizer  (proposed — not in the PR)
             elif resume_checkpoint_exists and megatron_cfg.checkpoint.load_optim:
-                if not _force_sync_model_from_optimizer_fp32(optimizer):
-                    _sync_model_params_from_loaded_optimizer(optimizer)
+                _force_sync_model_from_optimizer_fp32(optimizer)
     torch.distributed.barrier()

The first hunk is the same three lines that grpo.py and sft.py already have. The second keeps the PR's load_optim check and drops the extra sync. With it, _sync_model_params_from_loaded_optimizer and its three tests can be deleted. The review also asks to check each weight file once, not once per tensor, in row 2.

🟡 review comment — community_import.py:87 review comment — test_community_import.py:277

So what. Rows 1 and 2 are real fixes: SFT v2 now starts from the checkpoint you configured, and a conversion with a missing weight file is redone instead of reused. Row 3 changes no weights in any shipped config at these pins. It only adds a blocking all-gather to every Megatron resume, in GRPO, DPO and distillation too, not just SFT.