Which generation failures the run survives, and which end it

PR #3591 · checked against 3d9ce210 · see also what the review changed

Six pages, one question each — this one: which failures recover.

Two things decide the outcome, and you need both to predict it: what the ledger already thought of the shard when the weight sync began, and how the sync then went. There is no separate state for the refit. Its fate is written entirely in the shard's state, plus one hard rule — recover at most once per sync. Below is every combination.

The rule that explains most of the grid: the retry only happens if someone can be named. Recovery is two steps in order — work out which shard to drop, rebuild the group without it, and only then retry the refit. If the first step cannot name anyone, it raises and the retry is never reached. So a crash recovers because it identifies itself: Ray reports the dead process and the probe confirms it. A hang identifies nobody, and only recovers if the ledger already held a suspect. Retrying without shrinking would be worse than not retrying — the same full fleet, the same silent rank, the same hang.

On nccl_reshard there is a second requirement: the fault must land outside the bulk transfer. An abort mid-transfer orphans work on the trainers' own devices, and no rebuild on a device in that state can succeed — so naming the shard no longer helps. Since 9a87c0123 the run detects this and ends in seconds with a named cause instead of wedging. That turns both crash cells below into transport splits, not just the hang cell.
Shard state before the sync × how the sync went refit SUCCEEDS refit HANGS refit CRASHES HEALTHY nothing wrong that anyone had noticed normal step Nothing to do. run ends Abort fires, but nobody is absent and nobody was suspect. Nobody to drop, so no rebuild, so no retry. depends on transport The crash names the shard — but see the reshard caveat. broadcast → recovers reshard mid-refit → ends SUSPECT wedged engine: its own generations have been timing out. Still gets traffic, still refits. continues A wedged engine can still do NCCL. It keeps failing generations and reaches DEAD on its own. depends on transport The one suspect is condemned DEAD, group shrinks, retry. broadcast → continues reshard → ends (see below) depends on transport The dead process is visible, so no guessing — same split as the row above: broadcast → recovers reshard mid-refit → ends DEAD already condemned before this sync began never in the collective in the first place The reconcile at the top of the sync rebuilds over the survivors before a single byte moves, so this shard cannot hang or crash the refit. This is where a shard noticed between steps lands, and it is the cheap path. It stays out until #3592 restarts it: DEAD, then RESTARTING, then STALE, then re-admitted at a later refit.
Read a cell as: this is what the ledger thought, this is what then happened, so this is the outcome.

The same grid as a table, with what does the work

Gen workerRefitOutcomeWhat actually happens
HEALTHYsucceedscontinues The ordinary step. The reconcile at the top is a comparison, not a rebuild — each synchronizer remembers the membership it last built with.
HEALTHYhangsrun ends The watchdog aborts, but the probe finds every process alive and the ledger holds no suspect — so there is nobody to drop, no smaller group to rebuild, and the retry is never reached: recovery raises first. The code deliberately refuses to guess: "condemning the wrong one costs a healthy shard and leaves the real culprit in the rebuilt communicator — the same hang, one shard smaller." The error names the suspect list so you can see why it gave up.
HEALTHYcrashestransport RayActorError, then the post-abort probe sees the dead process, record_actor_death marks it DEAD, the group is rebuilt without it, and the refit is retried once. On packed broadcast that recovery completes. On nccl_reshard it never gets that far: the aborted bulk transfer has already orphaned work on the trainers' devices, sync_stream_within tags the abort, and the controller refuses the recovery and ends the run in seconds — see the box below. A death that lands between syncs recovers on both transports, because the shard is dropped before the next collective starts (the DEAD row).
SUSPECTsucceedscontinues Common and harmless. A wedged engine can still take part in a collective; it just cannot generate. It keeps timing out requests and reaches DEAD on its own, after which it lands in the DEAD row.
SUSPECThangstransport The abort is evidence that something stopped participating; SUSPECT says which. Exactly one suspect and condemn_silent_participant marks it DEAD, then reconcile(force=True) shrinks the group and retries. Works on packed broadcast; ends the run on nccl_reshard — see the box below. With two or more suspects, or none, it refuses and ends.
SUSPECTcrashestransport The process death is visible, so no attribution is needed. Same split as HEALTHY × crashes: broadcast recovers, reshard mid-refit ends the run.
DEADanycontinues Excluded before the collective started, so it cannot affect this refit at all. Stays out until #3592 restarts it.
Not about one generation shard
a trainer diescrashesrun ends No remediation, by design. Trainers are never excluded — dropping one would change the parallelism the model was sharded for. This stack is about generation shards.
a 2nd shard failsduring the retryrun ends Recover at most once: "a second failure is a real fault, not a membership problem, and retrying forever would recreate the wedge this exists to remove." The second shard is not even condemned, because the probe ran before the retry.
too few left—run ends raise_if_exhausted once serving shards drop below min_healthy_shards. The right outcome: a run that cannot generate should stop.
Why a mid-refit fault — wedged or dead — is recoverable on broadcast but not on nccl_reshard. Aborting a communicator does not retire work already sitting on a CUDA stream — and on the reshard path those orphaned sends are on the trainers' devices, not the failed shard's. So nothing you do to the remote rank retires them, and torch.cuda.synchronize() never returns. Job 6485245 measured it on 4×GB200: both policy workers parked in synchronize 1801 s after their own abort had already logged. The author eliminated every alternative by measurement: killing the shard outright (job 6523731 — trainers wedged anyway), pinning the release to the caller's CUDA device (job 6582457 — still stuck, with a SIGKILLed peer), and standing the deadline down on a confirmed death (lost the race). What landed instead (9a87c0123): sync_stream_within tags its abort with a second marker, the controller refuses to enter the recovery on that tag, and the run ends with a message saying this is the known limit of the transport — exit in ~10 s for a killed shard, ~90 s for a frozen one, against 1800 s wedges before. A plain abort without the tag still recovers, and a unit test pins that boundary.

Does it work?

The abort is measured: on 2×A6000, a survivor blocked in a collective was released 0.15 s after another thread called abort(). The reshard limit is measured too — job 6485245 for the stuck synchronize, and job 6523731 as the disproof of killing the shard.

What is not measured is the whole grid end to end in CI. The recovery harness needs 3 GPUs and both lanes have 2, so those variants skip; the rows above were exercised by hand on larger hardware. The lane now registers the reshard-plus-frozen corner and makes a skip visible instead of letting it read as a pass, so at least the gap is legible.

The short version. A shard that dies is handled, whenever it is noticed. A shard that wedges is handled if the ledger already doubted it and you are on the broadcast transport. The only case with no answer is a hang nobody saw coming — and refusing to guess there is deliberate, because condemning the wrong shard reproduces the same hang one shard smaller.