Why the refit group is rebuilt instead of shrunk

PR #3591 · links pinned at ab0ffe2f and nccl v2.30.7-1

Six pages, one question each — this one: the design choice.

NCCL can resize a communicator in place. The pinned version declares ncclCommShrink and ncclCommGrow, so "drop the dead rank and carry on" is a real option, not something the library refuses. This PR throws the communicator away and builds a new one instead. That looks like the more expensive choice, and the reason it is right has nothing to do with communicators — it is about a plan that lives beside them.

One parameter, split across 3 generation shards. Shard 0 dies. The reshard transport does not broadcast the same bytes to everyone — each shard receives its own slice. The plan, made once at setup slice A → shard 0 slice B → shard 1 slice C → shard 2 every destination derived from gen_world_size = 3 shard 0 dies Shrink the communicator only The group is now 2 ranks. The plan is untouched. slice A → shard 0 (nobody) slice B → shard 1 slice C → shard 2 Slice A is never written. No error. A stale plan is still a valid plan — the survivors come back holding weights nobody ever sent them. Rebuild, and regenerate the plan New group over the survivors, new destinations. slice A' → shard 1 slice B' → shard 2 gen_world_size = 2 Every slice has an owner that exists. The parameter is re-split for the fleet that is actually there.
Shrinking fixes who is in the group. It does not fix who each slice was addressed to.

Two decisions, forced by different things

These get run together in conversation, so it is worth separating them. Only one of them is a judgement call.

DecisionForced or chosen?Why
Break the hang with abort(), not destroy() Forced destroy is itself a collective that every rank has to call, which is exactly what a dead rank cannot do. abort ends in-flight work whether or not the peers are alive, so it is the only one that works on a path built for dead peers. See StatelessProcessGroup.abort.
Restore service by rebuilding, not shrinking Chosen NCCL offers both. The plan has to be regenerated on any membership change regardless, and once you are doing that, a second way to change membership buys nothing.

The deciding reason is the figure above. prepare_nccl_reshard_refit_info derives every parameter's destination from gen_world_size. Change the fleet and every destination changes with it. Shrinking the communicator alone does not reduce the number of receivers — it orphans the slices the dead shard owned, and the run continues with weights that were never written. Silently, which is the part that matters.

Two smaller reasons, both minor next to that one. Restart time is dominated by the minutes an engine takes to reload a model, so the rendezvous a shrink would save is noise. And because the rebuild shares _build(membership) with the very first build, every normal run exercises the recovery arithmetic — so the code that only matters after a failure is not code that only runs after a failure.

Does it work?

The half that is forced is the half that was measured. On 2×A6000, with a peer SIGKILLed mid-broadcast, a survivor blocked in the collective was released 0.15 s after another thread called abort(). That is the fact the whole design rests on: if aborting from a second thread had not freed a blocked collective, none of the rest would matter.

The half that is chosen has not been compared against the alternative — nobody has built the shrink version and measured it, and given the argument above nobody should need to. What has not been shown end to end is the rebuild itself under CI: the recovery harness needs 3 GPUs and both lanes have 2, so those variants skip. The rebuild has been exercised by hand on larger hardware.

One thing the shrink-versus-rebuild framing hides. Both options assume the abort actually freed every rank. On the reshard path the transfer also creates split child communicators, and NCCL does not pass an abort down to them — so a rank blocked on a child stays blocked whichever option you pick. That is raised on the PR, and it has to be fixed before either design is safe.