What a shard restart can and cannot do when a whole node dies

Companion illustration for NVIDIA-NeMo/RL #3592. Permalinks pin the PR head 5b4a732; Ray behaviour is quoted from the pinned version, 2.56.1.

Seven pages on this PR, one question each — this one: the failure the restart is not built for.

The restart puts the replacement actors back on the same placement-group bundle the dead ones held. That is deliberate — Ray does not release a bundle when an actor dies, so the replacement lands on the same GPU. It also means the restart is only ever as good as the bundle. If the engine died, the bundle is still there and the restart works. If the node died, the bundle went with it, and everything below follows from that.

Where a replacement bundle would have to go

A job with 2 NVLink domains and segment_size: 3, so nodes are handed out in complete 3-node segments. Training takes one segment, generation takes two — 9 nodes, which is what the job asked for, so nothing is spare. Node 7 dies.

What the restart does then

It does not fail. It stops, quietly, in a place with no timeout. Reading the calls in order:

recreate_worker(8) ray.kill(old) # already gone with the node create_worker(pg, bundle_idx=8) # .remote() returns a handle straight away — # Ray does not wait to see if it can be placed -> returns a handle to an actor that will never start post_init_async(leader) ray.get(...) # blocks. the actor is still PENDING_CREATION, # so this call has nothing to wait on but forever

Ray creates actors asynchronously: create_worker ends in .remote() and hands back a handle whether or not the bundle can be scheduled. So the restart gets all the way to post_init before anything notices, and that ray.get has no deadline.

Three things follow, and none of them is loud:

The shard stays RESTARTING, which counts as absent, so training carries on without it. That part is fine and intended.
The attempt is never marked failed, so max_restart_attempts_per_shard is never spent and the shard is never RETIRED. It sits in the in-flight set for the rest of the run, so no later tick retries it.
The parked call is running inside asyncio.to_thread, whose workers are joined at interpreter exit — so it can also hold up shutdown at the end of the run.

There are no spare nodes, and nothing plans for them. The only "spare" in the codebase is a pool of spare prompts for the data plane. At the cluster level the job asks for exactly train_nodes + inference_nodes, and whatever is left over is handed to the teacher model rather than held back. So on a fixed allocation with no autoscaler — a Slurm job — a lost node is a lost node.

Making node loss survivable is a real feature, not a tweak: you would have to allocate spare nodes up front, keep them out of every cluster, and pick a replacement inside the same NVLink domain when segment_size is set. Worth saying plainly that today the restart handles a dead engine, not a dead machine.