Why a restarted engine comes back already halfway condemned

Companion illustration for NVIDIA-NeMo/RL #3592. Permalinks pin the PR head 5b4a732.

Seven pages on this PR, one question each — this one: one bug, start to finish.

The health ledger keeps a small record per generation shard: what state it is in, how many requests it has failed in a row, how many restarts it has used. When a shard is restarted, the process is destroyed and a new one takes its place, so most of that record is about something that no longer exists. mark_loaded is where the ledger is told the replacement is up, and it clears two of the counters, with the reason written next to them: "carrying that count into a fresh engine would let a single unlucky probe re-condemn it immediately, burning a restart attempt for nothing." That reason is right. It also applies to two fields it does not clear.

shard 1 fails 2 requests report_failure a refit aborts mark_weights_partial it is condemned condemn_silent_ participant restart finishes mark_loaded next refit lands report_refit state SUSPECT STALE DEAD STALE SUSPECT state_before_partial none SUSPECT SUSPECT SUSPECT read, then cleared consecutive_reported_failures 2 2 2 2 2 (not reset) restart_attempts 0 0 0 1 of 5 1 of 5 base_url the old port the old port the old port the new port the new port state_before_partial is not a history of the state row above. Only the second column ever writes it, and after that it just sits there. At the fourth column the process is destroyed and a new one takes over. mark_loaded picks up the new port and clears the two probe counters, then leaves the rest. So the next completed refit reads a verdict, and a failure count, that belong to a process that no longer exists. still the dead engine's — should have been cleared correctly updated for the new process

Shard 1 of four. The thresholds are the shipped defaults: three failed requests in a row condemn a shard, and a shard gets five restarts before it is retired for good.

"Partial" is about the weights, not the state. A refit is a weight transfer from the trainer to the engines. If one is aborted halfway, every engine that was receiving is left holding a mix of old and new weights — a partly-applied update. mark_weights_partial is what records that, by moving the shard to STALE so it stays in the next refit but serves nothing meanwhile.

So state_before_partial means "what this shard was before the abort pulled it out of service". It is written in exactly one place — that function, and only when the shard is not already STALE — so it never records STALE itself. That guard is deliberate: the field exists so a completed refit can put the shard back where it was, and "back to STALE" would say nothing.

It never decides whether a shard joins a refit. That is decided by the state on its own: DEAD, RESTARTING and RETIRED are treated as absent and left out; STALE is not. A STALE shard takes part in the transfer normally — that is the only way it stops being stale. This field is read once, at the very end of a completed refit, and only to pick which state the shard goes back to.

And it is not a shortcut back to HEALTHY — it is what makes that harder. Without it a suspect shard would go SUSPECT, then STALE when a refit aborts, then straight to HEALTHY when the next one lands, and its failure count would be wiped by a refit that had nothing to do with why it was failing. That count is the only thing that can catch an engine which answers health checks and still fails every request, so wiping it means such an engine is never caught. The field keeps the suspicion alive across the abort.

That is why the third column still reads SUSPECT. condemn_silent_participant moves the shard to DEAD and does not touch this field at all. Only one other line ever changes it: report_refit, which reads it and clears it in the same breath. Nothing in between — not the restart — resets it, which is the whole bug.

This is guaranteed on that path, not unlucky timing

It would be easy to read the figure as a rare interleaving. It is not — on the path that condemns a shard for going silent inside a weight transfer, the third column always leaves state_before_partial set to SUSPECT.

That is because suspected_shards() is defined as "shards that are STALE and whose state_before_partial is SUSPECT", and the recovery picks its culprit from exactly that list. The test that chooses the shard is a test that the field is set, so every shard condemned this way carries it into DEAD. Nothing between there and the next refit ever clears it.

Run against the real module, with no Ray and no GPU:

2 failed requests -> SUSPECT state_before_partial=none streak=2 a refit aborts -> STALE state_before_partial=SUSPECT streak=2 condemned -> DEAD state_before_partial=SUSPECT streak=2 restart finishes -> STALE state_before_partial=SUSPECT streak=2 <- new process refit completes -> SUSPECT streak=2 <- not HEALTHY one failed request-> DEAD

A brand-new engine that loaded cleanly and completed a full weight transfer comes back SUSPECT, holding the dead engine's failure count, one request away from being killed again. report_refit takes an early return on that branch, so it skips the reset that would have zeroed the count.

The fix: a new process has no history

The first three events are the same either way: the shard really was failing requests, and it really was the right one to condemn. Everything below is about the last two — what the restart hands to the refit, and what the refit does with it.

TODAY — as the PR is written restart finishes · mark_loaded next refit lands · report_refit state STALE SUSPECT state_before_partial SUSPECT — still the dead engine's read, then cleared consecutive_reported_failures 2 — carried over 2 — the reset is skipped one more failed request condemns it: DEAD, a second restart, another of its five attempts gone WITH THE TWO LINES ADDED restart finishes · mark_loaded next refit lands · report_refit state STALE HEALTHY state_before_partial none — cleared with the port nothing to read consecutive_reported_failures 0 — cleared with the port 0 one more failed request takes it to SUSPECT, which is where an engine with one failure belongs

The edit that gets there:

Both lines go in mark_loaded, in the block it already uses to clear the probe counters — after line 426, before the transition on line 427. Nothing else moves.

# fleet_health.py, mark_loaded — today 425 shard.consecutive_probe_failures = 0 426 shard.consecutive_probe_successes = 0 427 self._transition(shard, ShardState.STALE) # -> next refit: SUSPECT, streak 2
# the same function, with the two lines a replacement process needs 425 shard.consecutive_probe_failures = 0 426 shard.consecutive_probe_successes = 0 + shard.consecutive_reported_failures = 0 # the dead engine failed those requests, not this one + shard.state_before_partial = None # that verdict was about the dead engine too 427 self._transition(shard, ShardState.STALE) # -> next refit: HEALTHY, streak 0

The bug in one line: a guard built to survive one aborted refit also survives a new process. Keeping a suspicion across an abort is right — the engine is the same one, and it is still suspect. Keeping it across a restart is not: that engine is gone, and the record outlives it.

What this costs. One spurious condemnation, and it is not cheap: the shard is killed and reloaded again — minutes with a quarter of the fleet down — and one of its five restarts is spent, all triggered by a single unrelated failed request. It does not repeat, because report_refit clears the field as it reads it, so the shard is correct from the second cycle on. But it is a real restart of a healthy engine, and it makes the restart budget a misleading number: a shard can be two restarts from retirement having only genuinely failed once.