Two knobs with almost the same name: which one actually packs?

NVIDIA-NeMo/RL #4105 — all links pinned to f31f793.

Four pages on this PR — this one: what every packing config key does, and how to set them.

Energon SFT packing is configured from two places at once, and two of the keys are named almost identically: data.energon.max_samples_per_sequence and policy.sequence_packing.max_sequences_per_bin. Both read as "max <thing> per <other thing>" using the same two words with the meanings swapped. Only one of them packs.

Both count samples, and a sample is one whole conversation — system prompt, user turn, assistant turn, a tool call and its reply, all of it. Turn count does not matter and token count only matters for the second knob. The figure below starts by pinning that down, then shows what each knob changes.

The two that get confused

FIRST: what counts as one sample Both knobs count whole conversations. One conversation is one sample, however many turns it has. system user assistant tool assistant five turns, one sample one block = this Below, this same conversation is drawn as a small block. Its colour says which file it was read from. 1 · data.energon.max_samples_per_sequence — the ORDER samples arrive Colour = which shard file on disk the conversation was read from. Nothing else. shard 1 shard 2 shard 3 A shard is one file the dataset is split across. Each block inside it is one conversation, exactly like the five-turn one drawn above. unset shards interleave, one conversation at a time = 4 four in a row from one shard, then move on Same twelve conversations either way. Only the arrival order changes — so it changes shuffling, not layout. 2 · policy.sequence_packing.max_sequences_per_bin — how many samples share a pack Same five conversations, still coloured by the shard they came from. The packer does not care which. unset all five in one pack — only the token budget stopped it = 2 never more than two, so three packs instead of one Three packs means three forward passes where there was one. This is the knob that changes layout. Legend from shard 1 from shard 2 from shard 3 one pack (the row the GPU sees) Block width is the conversation's token count, so a wider block is a longer conversation. Neither knob is required. Unset means: Energon's own read order, and packs limited only by tokens.

The whole set, and what each one does

KeyRequired?What it does
data.energon.packing_buffer_sizeyes The master switch. Non-null is what makes setup treat this as an Energon-packing run. Also how many samples Energon buffers before choosing bins.
policy.sequence_packing.enabledyes Must be true, or setup rejects the config.
policy.sequence_packing.fuse_lossyes Must be true, same check.
policy.sequence_packing.algorithmyes Which packer. Any of the six; the recipes use balanced_greedy_knapsack.
data.max_input_seq_lengthyes The pack width. Passed to the packer as its bin capacity, and every pack is padded out to it.
policy.make_sequence_length_divisible_byyes Each conversation is padded up to a multiple of this before packing. Must divide max_input_seq_length and match the parallelism — checked at startup.
policy.sequence_packing.max_sequences_per_binno Caps conversations per pack. Unset means only the token budget limits it.
data.energon.max_samples_per_sequenceno Shard read order. Nothing to do with packing.
policy.dynamic_batching.enabledmust be off Rejected together with Energon packing.
policy.sequence_packing.train_mb_tokensunclear See the note at the bottom — the type says required, the packing path does not appear to use it, and no recipe sets it.

How to turn it on

The smallest config that satisfies every check, with the two optional knobs shown for contrast:

policy: sequence_packing: enabled: true fuse_loss: true algorithm: balanced_greedy_knapsack max_sequences_per_bin: 16 # optional: cap conversations per pack make_sequence_length_divisible_by: 16 data: max_input_seq_length: 4096 # becomes the pack width energon: packing_buffer_size: 64 # the master switch max_samples_per_sequence: 16 # optional: shard read order, NOT packing
The short version. Energon does the packing, but the packer is configured under policy.sequence_packing — so you set keys in both blocks. The only key that controls how many conversations share a pack is max_sequences_per_bin. The similarly named max_samples_per_sequence changes what order samples come off disk in, which affects shuffling, not layout.
One loose end. train_mb_tokens is declared a required key of SequencePackingConfig, and _packing_args reads it with a plain subscript whenever sequence packing is on. But the pack width on this path comes from max_input_seq_length, and none of the three new recipes set train_mb_tokens — nor does anything in their defaults: chain. Either something outside the chain supplies it, or it is unused here and the type should say so. Asked on the PR; this page will be updated with the answer.