Why each Single Controller rollout attempt now gets its own Gym task number

NVIDIA-NeMo/RL #4245 Β· all RL links pin head fc77827, Gym links pin 267305e.

Before
Single Controller sent Gym whatever _ng_task_index the dataset row had, or none. The same prompt seen again (next epoch, or a re-run) sent the same number again.
This PR
A counter gives every rollout attempt a new number. The counter is saved in checkpoints and snapshots, so a resumed run carries on from the saved value.
Status
Fixed by this PR within one run. No correctness bugs found in review. One small comment asks the code to say why the number must be fresh.
Background. In Single Controller, RL turns each prompt into a few rows and sends them to a Gym agent. Every row carries a task number (_ng_task_index) and a rollout number, 0, 1, … inside the group. When RL sends no explicit rollout ID, Gym builds a key for the rollout from those two numbers: "{task}-{rollout}". Dotted words have a short explanation: hover or tap them.

Gym builds the key in maybe_rollout_id_from_run_body, and its docstring warns: β€œReused task and rollout indices produce a repeated capture key” (L48). With observability on, Gym files /verify calls under that key (server_utils.py:514). The Harbor agent names its job folder t{task}-r{rollout} and returns any finished trial already in it (app.py:139, L518-L540). So a task number that repeats makes a new rollout look like an old one.

One prompt, seen twice, two rollouts each

Dataset prompt P has its own _ng_task_index: 7. Two rollouts per prompt (rollout numbers 0, 1). Before (main) epoch 1: send P {_ng_task_index: 7} Gym keys "7-0", "7-1" epoch 2: send P {_ng_task_index: 7} Gym keys "7-0", "7-1" βœ— same keys as epoch 1 observability files both epochs under one key; Harbor finds the epoch-1 trial and returns it instead of running the new weights. No task number at all β†’ no key; Harbor rejects the row. After (this PR) epoch 1: counter β†’ 40 {_ng_task_index: 40} Gym keys "40-0", "40-1" epoch 2: counter β†’ 53 {_ng_task_index: 53} Gym keys "53-0", "53-1" βœ“ every attempt has its own keys run_rollout takes a number: rollout_manager.py:1015 stamped on every row: rollout_manager.py:1094 Checkpoint after epoch 2 saves next = 54: trainer save: single_controller.py:4539 snapshot save: single_controller.py:3892 resume restores 54: single_controller.py:426 key repeats unique

How the fix works

Where the fix lands: RL only. rollout_manager.py hands out the numbers; single_controller.py, grpo.py, rollout_checkpoint.py and setup.py save and restore the counter. No Gym change: Gym already uses whatever number RL sends.

@@ rollout_manager.py L983-L986 Β· new method on AsyncNemoGymRolloutImpl
+    def _take_nemo_gym_task_index(self) -> int:
+        task_index = self._next_nemo_gym_task_index
+        self._next_nemo_gym_task_index += 1
+        return task_index
@@ rollout_manager.py L1013-L1016 Β· run_rollout β€” one number per attempt
         rollout_inputs = self._build_inputs(
             input_sample,
+            task_index=self._take_nemo_gym_task_index(),
             rollout_ids=rollout_ids,
@@ rollout_manager.py L1090-L1094 Β· _build_inputs β€” every sibling row shares it
         template_row: dict = copy.deepcopy(input_sample["extra_env_info"])  # type: ignore
+        # One fresh ID per run_rollout attempt, shared by all sibling rows.
+        template_row[NEMO_GYM_TASK_INDEX_KEY] = task_index
@@ single_controller.py L4538-L4541 Β· _save_checkpoint_impl β€” save next value
             save_state.sampler_dispatch_index = self._sampler.dispatch_index
+            save_state.next_nemo_gym_task_index = (
+                self._rollout_manager.get_next_nemo_gym_task_index()
+            )
@@ single_controller.py L425-L428 Β· __init__ β€” restore it on resume
         self._rollout_manager = actor_args.rollout_manager
+        self._rollout_manager.set_next_nemo_gym_task_index(
+            actor_args.save_state.next_nemo_gym_task_index
+        )

In order: each call to run_rollout takes the next number (40, then 53 in the example). _build_inputs writes it on a copy of the row, so the dataset sample keeps its own 7. A retry of failed rows inside one attempt reuses the built rows, so it keeps 40 (tested at test_rollout_generation_failures.py). The saved value, 54, also goes into rollout-only snapshots (L4116). Old checkpoints load as 0, and old snapshots keep the trainer's value.

The older async path uses the same key with a different meaning: there, _ng_task_index is the prompt's place in the dataloader (trajectory_collector.py:617). The two paths never share rows, so nothing breaks. The review asks for the comment in _build_inputs to say why Single Controller needs a fresh number, and that it is not a dataloader position.

🟑 review comment β€” rollout_manager.py:1091

So what. Within one run, Gym no longer sees two rollouts with the same key, so a Gym cache keyed on the task number (like Harbor's job folders) cannot hand back an old result for a repeated prompt. The counter is per checkpoint: numbers handed out after the last save are handed out again after a crash, and a fresh run starts at 0. That only matters to a Gym agent that caches results in a folder shared across runs (Harbor today). No NeMo-RL recipe uses Harbor, so this is a possible Gym follow-up, not an ask for this PR.