Three places NeMo-RL can pack SFT data, and where they all meet

spans #3380 (offline prepacked, open) and #4105 (Energon, open) — current-main links pinned to f31f793, #3380 links to b84eba1.

The map across both PRs — this one: every way a packed row can be produced, and the single place they converge.

Packing puts several short conversations into one long row so the GPU is not multiplying through padding. NeMo-RL has one way to do this today and two more proposed in open PRs. They differ in when the boundaries are decided — at dataset build, at data load, or at the training step — and in how much padding that decision leaves behind.

The map below carries the same two conversations through all three: A is 5 tokens, B is 10. Watch the third column — every path produces the same real boundaries and a differently sized row.

The map

WHAT IT STARTS FROM (the same two conversations every time) WHERE IT PACKS THE ROW IT HANDS ON 1 OFFLINE #3380 not on main conv A, 5 tokens conv B, 10 tokens loose conversations, before any run a preprocessing script writes the packed row to disk 16 wide, chosen when written cu_seqlens [0, 5, 15] — stored in the row re-read identically every epoch 2 LOADER #4105 not on main conv A, 5 tokens conv B, 10 tokens the same two, reached by streaming a buffer the Energon loader packing.py:89 tail pad 36 wide cu_seqlens [0, 5, 15] padded [0, 8, 36] always the full max_input_seq_length 3 TRAINER on main today conv A + padding conv B the same two, already a padded 2 x 10 batch the policy worker, per step megatron/data.py:1284 20 wide, sized to this batch cu_seqlens [0, 5, 15] padded [0, 8, 20] rebuilt every step, discarded after All three agree on the real boundaries — cu_seqlens [0, 5, 15]. They disagree only on the padding. PackedSeqParams(qkv_format="thd") one flat token run + a boundary list Reading the row conversation A, 5 tokens conversation B, 10 tokens padding between them tail padding, path 2 only Bar width is token count, so the three output rows are drawn to the same scale: 16, 36 and 20 tokens wide. Path 1 picks its width once. Path 2 always uses the configured maximum. Path 3 uses only what this batch needs.

Side by side

 1 · Offline prepacked2 · Loader packs3 · Trainer packs
Status#3380, open#4105, openon main
Boundaries decidedat dataset buildas the loader buffersat the training step
Where the work runsCPU, once, offlineloader workerGPU worker, every step
Same row each epoch?yesnono
Row widthfixed when writtenalways max_input_seq_lengthas wide as the batch needs
Rows per microbatchoneone packseveral
Built forreproducing a Megatron-LM baseline exactlymultimodal SFT streamed through Energongeneral use
Detail pagetwo packerswho packscovered in both

Why they can converge at all

Because the thing the model needs is small: a flat run of tokens, and a list saying where each conversation ends — the shape the Energon path builds and the trainer path builds. Everything upstream is free to decide those two however it likes.

PackedSeqParams( cu_seqlens_q = ..., # where the real tokens end cu_seqlens_q_padded = ..., # where each padded slot ends qkv_format = "thd", # flat tokens, not a rectangle of rows )

The attention kernel reads cu_seqlens to know where one conversation stops and the next starts, so it never lets them see each other. Whether that list was computed months ago by an offline script, a second ago by the loader, or right now by the trainer makes no difference to it.

Reading the map. Two of these three are proposals. Only the trainer-side path is on main today, so a reader hitting this page from a search should not assume the other two are available — #3380 and #4105 were both still open when this was written. What the map is good for is seeing that they are not competing designs: they pick different moments on the same timeline, and they agree on the handoff.
The one place a difference survives the merge. Paths 1 and 3 hand the kernel the real boundaries in cu_seqlens_q and the padded ones separately. Path 2 passes the padded array for both, on purpose — some consumers read cu_seqlens_q as a wrap-around point, and real boundaries would let a value roll into the padding. So pad_between_seqs is False there and True on the others. Same object, different contents, and the reason is worth knowing before adding a fourth path.