How a shard gets marked healthy by a refit that never sent it any weights

Companion illustration for NVIDIA-NeMo/RL #3592. Permalinks pin the PR head a815a8cb.

Seven pages on this PR, one question each — this one: one bug, start to finish.

Before
The promotion asked each shard only "are you STALE?", so a shard that finished restarting in the middle of a refit was marked healthy holding the checkpoint it read off disk.
This PR
Adds participants — the shards eligible for this refit — and promotes only shards that are STALE and in that set.
Status
Partly fixed. The guard landed and closes the wide version of this. It builds its set from a second read of the fleet rather than from the set the communicator was built over, so a narrower window is still open — raised in review, not yet addressed.

A generation engine that restarts comes back holding whatever weights it read off disk at startup, so the ledger parks it in STALE: the process is alive and can join a weight transfer, but it is not allowed to answer requests. The module docstring is firm that a completed weight transfer — a refit — is the only way out of STALE. The code that does the letting-out, _record_refit_landed, used to ask each shard only "are you STALE?" and never "were you in the group I just sent weights to?" Before this PR those two questions always had the same answer. This PR adds a way for a shard to turn STALE in the middle of a refit, so it now asks both — and the figure below is the case that still slips through.

One refit, and the four moments shard 2's restart can finish Shard 2 died earlier. Its reload takes minutes and nothing waits for it, so it can land anywhere. membership read · L3443 participants read · L4705 promote · L4745 W1 W2 W3 W4 communicator rebuild weight transfer the shard is RESTARTING until its reload lands, then STALE; the promotion is what makes it HEALTHY What each window means W1 — lands before the membership read: already STALE, so it joins the refit and really gets weights. W2 — lands during the rebuild: left out of the communicator, but counted as a participant. W3 — lands during the transfer: left out of both. W4 — lands after the promotion: sits out this refit entirely, and the next one takes it as a W1. always correct still wrong today was wrong, fixed by this PR

Only W2 and W3 were ever wrong. Both were wrong before this PR; the PR fixes W3 and leaves W2. "Promoted" means the ledger moves the shard from STALE to HEALTHY — it starts answering rollout requests again.

The reload finishes…In the communicator?
(membership read, L3443)		In participants?
(second read, L4705)		Gets weights?BeforeAfter this PRWith the fix
W1 before the membership readyesyesyescorrectcorrectcorrect
W2 during the rebuildnoyesnopromotedpromotedskipped
W3 during the transfernononopromotedskippedskipped
W4 after the promotionnononocorrectcorrectcorrect

W2 is the odd row, and the reason is the two reads. The communicator is built from the membership read; participants comes from a second read taken after that one has already awaited. A reload landing between them makes the shard STALE — and STALE is not absent — so it is in the set the promotion trusts while the communicator was built without it.

Why nothing catches it

There is a metric that looks like it was built for exactly this. as_metrics publishes a per-shard weight version with the comment "Catches a STALE shard that somehow served: a serving shard whose weight version trails the trainer is a correctness bug, not a capacity problem."

It cannot catch it, because the number is written by the promotion rather than by the transfer. report_refit L476 is the only line in the repo that assigns it, and report_refit is only ever called from the promotion, only for shards that are STALE. So a shard that has never been restarted never goes through it and reads 0 forever, however many refits it has received — and the one shard that got nothing reads the current version, because the promotion is what put it there. The metric says the opposite of the truth in both directions.

What shipped, and where it still leaks

The PR takes the right shape. It computes the eligible set once, before the transfer, and promotes against it:

# single_controller.py L4705 — before the transfer participants = self._refit_participants() # ... weight transfer ... self._record_refit_landed(participants) # STALE *and* in participants

The comment above it names the hazard exactly: "a restart takes minutes and nothing blocks it, so mark_loaded can turn a shard STALE mid-transfer." Right idea. The gap is where participants gets its answer.

# _reconcile_refit_membership — the communicator is built from THIS read absent = self._gen_fleet.absent_shards() # shard 2 absent -> excluded await asyncio.to_thread(reconcile_communicator, absent, force) # ^ this await yields the loop. mark_loaded runs here. shard 2 -> STALE # back in _sync_weights — participants asks the fleet AGAIN participants = self._refit_participants() # reads absent_shards() a second time # shard 2 is STALE now, and STALE is not absent -> shard 2 IS in participants

So the two sets come from two different reads with an await between them. A reload landing in that gap puts shard 2 in participants while the communicator was built without it — and the promotion lets it out on the strength of a transfer it never received.

The fix — proposed, not in the PR

Nothing below is implemented. The right set is the one the reconcile already used. Have _reconcile_refit_membership return the absent set it reconciled against, and have _refit_participants take that set instead of calling absent_shards() itself. Then one read feeds both, and there is no gap to land in.

Both call sites need it — the main one at L4705 and the recovery re-snapshot at L4739, where the rebuild happens inside the recovery and participants is read after it returns. By the code's own note the recovery one is the likelier of the two: the shard is restarting because that refit just failed.

What this costs. The shard serves training rollouts generated from an old checkpoint, and nothing reports it — no error, no crash, no metric. Rollouts from stale weights are still valid-looking text, so the run keeps going and the numbers quietly get worse. The module docstring names this as the exact outcome the design exists to prevent: "re-admitting it because it started answering probes again would feed training rollouts generated from a checkpoint hundreds of steps stale — invisible, and far worse than the outage that caused it." The rule is right; the promotion just does not enforce it.