What actually gets recreated when a generation shard restarts

Companion illustration for NVIDIA-NeMo/RL #3592. Permalinks pin the PR head 5b4a732. vLLM is pinned at 0.24.0 on x86_64 and 0.25.1 on aarch64.

Seven pages on this PR, one question each — this one: how far down the stack a restart reaches.

A generation shard is one vLLM model replica. NeMo-RL gives it tensor_parallel_size × pipeline_parallel_size Ray actors, one per GPU bundle, but only the first of them loads a model: that one gets bundle_indices, which is exactly what makes it the model owner. The rest set self.llm = None and return immediately — they exist to hold their GPU bundle. The owner builds the vLLM engine with distributed_executor_backend="ray" whenever TP×PP > 1, and hands it VLLM_RAY_BUNDLE_INDICES so vLLM's own Ray workers land on those same bundles.

One replica across two nodes

Take GB200 with 4 GPUs per node and tensor parallel = 8. One replica no longer fits on a machine, and NeMo-RL notices: model_parallel_size > cluster.num_gpus_per_node switches the cluster to a single unified placement group covering every inference GPU, instead of one placement group per node.

The bundles in that unified group are flattened into topology order and sliced into consecutive chunks of TP×PP. Each chunk is one replica. With 3 replicas that is 6 nodes and 24 bundles, and each replica straddles a node boundary — the group is only ever labelled with the node of its first bundle.

3 replicas · TP = 8 · 4 GPUs per node · 24 bundles across 6 nodes · replica 2 dies replica 0 — bundles 0-7 replica 1 — bundles 8-15 replica 2 — bundles 16-23 node 0 node 1 node 2 node 3 node 4 node 5 every replica sits on two nodes — 4 bundles on each replica 2, in full recreated all 8, across both nodes node 4 — bundles 16-19 node 5 — bundles 20-23 16 17 18 19 20 21 22 23 only this one builds the engine vLLM's own Ray workers ×8 placed on those same 8 bundles, so they span both nodes too model owner — has bundle_indices, builds the engine holds a GPU bundle, llm = None created by vLLM, not by NeMo-RL the replica is one slice of the flat bundle list; it is tagged with the node of its first bundle only, so nothing about it is node-local vllm_generation.py L475 — groups.append((node_idx[first_node], slice_)) a replica that straddles two NVLink domains logs a warning, because the collective then falls back to slower links

So with TP=8 on 4-GPU nodes, restarting replica 2 recreates eight NeMo-RL actors — four on node 4 and four on node 5 — even though seven of them never held a model. They have to come back because they hold the bundles that vLLM's own workers are placed onto.

The call chain, line by line

Every step of a restart, in order, with the code that does it.

#What happensCode
1The fleet-probe pump calls tick() after each probe round single_controller.py:2283
2tick() starts a restart for every shard that needs one, and returns engine_supervisor.py:69
3Picks shards that are DEAD and not already being restarted engine_supervisor.py:79
4mark_restarting → RESTARTING, which also spends one attempt from the budget engine_supervisor.py:91
5Spawns a background task — this is what makes it asynchronous engine_supervisor.py:101
6Hands the blocking work to a thread so the event loop stays free engine_supervisor.py:114
7restart_shard looks up the replica's first worker vllm_generation.py:715
8Loops over all model_parallel_size workers — the 8 in the figure vllm_generation.py:717
9Kills the old actor first, so it cannot still be holding the GPU worker_groups.py:727
10Gives the replacement a fresh name, because a dead actor's name lingers in the GCS worker_groups.py:736
11Replays the recorded creation call — same placement group, same bundle worker_groups.py:740
12Creates the actor. Ray returns a handle straight away, before it is placed worker_groups.py:227
13Swaps the new handle into the worker group in place worker_groups.py:749
14load_model, but only on the deferred-load path vllm_generation.py:744
15post_init on the owner rebuilds the engine — vLLM then makes its own 8 workers vllm_generation.py:748
16Asks the new engine for its base URL — a new port, not the old one vllm_generation.py:752
17mark_loaded records that URL and moves the shard to STALE engine_supervisor.py:128
—If any of that raises, mark_restart_failed puts it back to DEAD to be retried engine_supervisor.py:121

Steps 9 to 13 are the only ones that touch Ray actors. There is no step that reaches into vLLM to restart a tensor-parallel rank: those workers are vLLM's, made when the engine is built at step 15, so they return as a consequence rather than as an instruction.

Why not just let Ray restart the actor?

Ray can do this for you: max_restarts=-1 makes it recreate an actor forever. NeMo-RL uses that in several places — the environment actors, and the generation router. It is deliberately not used for generation workers, which take Ray's default of 0 — "the actor doesn't need to be restarted". That is what makes a RayActorError conclusive: nothing is going to bring that actor back on its own, so the probe can go straight to DEAD without waiting to see if it recovers.

The router's own docstring says why the difference exists:

max_restarts=-1 is only meaningful because of that: Ray recreates a restarted actor through __init__ alone, so a class that starts its server from a separate method comes back permanently broken.

That is exactly a generation worker. Ray would re-run __init__ and stop there — no post_init, no URL report, no place in the refit collective, and no way for the ledger to learn that any of it happened. You would get an actor that looks alive and serves nothing. Steps 14 to 17 in the table above are precisely the work Ray would skip, which is why the restart has to be driven by the application.

How far up does it restart? Up to and including every Ray actor NeMo-RL made for that one replica — all 8 in this example, on both nodes — and no further. The cluster, the unified placement group, its 24 bundles, the worker group and the other two replicas are all reused untouched. That is the point: a dead actor does not release its bundle, so the replacement lands on the same GPUs.

The one piece that can survive the kill. vLLM runs its EngineCore as an ordinary child process — no setsid, not a daemon — and every cleanup path vLLM and Ray provide runs inside the process being killed. So a SIGKILLed worker leaves the EngineCore behind, still holding its GPU, and the replacement then has nowhere to load. That is what RAY_process_group_cleanup_enabled=1 is for: it puts each worker in its own process group so the raylet can kill the leftovers. It has to be set before ray start, because the raylet reads it — setting it in the driver does nothing on a multi-node job, which is exactly the shape this page is about.