Why setting the failure tolerance to 0 makes a broken dataset hang instead of fail

Companion illustration for NVIDIA-NeMo/RL #4108 — permalinks pinned to d044f40, library behaviour read from megatron-energon 7.4.0, the version pinned in uv.lock.

Before
A run against a systematically broken dataset stopped quickly, with an error naming the bad sample.
This PR
Changes the failure tolerance from 1 to 0, which Energon reads as "stop counting failures".
Status
Open — not fixed in this PR. The change is still present at d044f40. Raised as a 🔴 must-fix comment on the PR.

Three pages on this PR, one question each — this one: one changed number and what it switches off.

Energon, the dataset library underneath this loader, counts how many samples fail to decode in a row. Past a limit it gives up and raises, so a run against a broken dataset dies quickly with an error. That limit is one number on the task encoder. This PR changes it from 1 to 0 at generic_sft.py:259 — and in Energon, 0 does not mean "give up immediately". It means stop counting. The same commit deletes the three-line comment that said so.

What the deleted comment said, and what 0 does

At the base commit these four lines sat together (base, where they still exist):

# Energon reads 0 as "disable tolerance checking" (megatron/energon/errors.py), # which would let a systematically broken dataset retry forever. 1 fails on the # first bad sample; raise it to tolerate transient decode errors. __default_failure_tolerance__ = 1

The comment is right. In megatron-energon 7.4.0, ErrorContext documents the field as "Maximum number of consecutive failures before raising FatalSampleError. Set to 0 to disable tolerance checking", and the raise is guarded by if self.tolerance > 0 and self._consecutive_failures >= self.tolerance:. At 0 that test is never true. Energon's own default for this field is 100.

A dataset where every sample fails to decode — wrong prompt_format, or the wrong processor sample attempts, left to right fail count 1 fail count 2 fail count 3 fail count 4 … forever handler prints a traceback and returns, so the loop just continues before this PR — tolerance = 1 raised FatalSampleError here, at the first failure the run stopped with an error you could read after this PR — tolerance = 0 compact_sample_error_handler() prints, returns, counter climbs, nothing ever raises The same number switches off a second stop, in the packing loop pre_packer_failure_tolerance > 0 also guards "did not yield any packs after N rounds" — dead at 0, so if the pre-packer returns nothing, its while-loop spins with no round limit either.

The fix

Put the number back above zero, and put the comment back with it. Energon's own default is 100, which is also what the deleted comment suggested — tolerate a run of transient decode errors, but still stop on a dataset that is simply wrong.

# Energon reads 0 as "disable tolerance checking" (megatron/energon/errors.py), # which would let a systematically broken dataset retry forever. 1 fails on the # first bad sample; raise it to tolerate transient decode errors. __default_failure_tolerance__ = 100

The three values worth keeping apart: 1 was the old setting, stop at the first bad sample. 100 is Energon's default, stop after a hundred in a row. 0 is not between them — it is off the scale in the other direction, and never stops at all.

What this costs if it ships. A misconfigured run — wrong prompt_format, wrong processor, a dataset path pointing at the wrong shards — no longer fails. It sits at step 0 printing tracebacks, holding its GPUs, until someone notices and kills it. The larger the job, the more expensive that silence is. Nobody has run this code path: no unit or functional CI job executed on this commit, and the review that found this had no GPU, so this is read from the library source rather than from an observed hang.