Sequential few-shot selection mutates the task's shared few-shot pool in place, corrupting later variance seeds

Start in src/lighteval/tasks/prompt_manager.py by reading _init_fewshot_sampling_sequential and comparing it with _init_fewshot_sampling_random. Run the reproduction from the issue and review the available regression test; done means each variance seed uses the original pool independently and task.fewshot_docs() remains unchanged.

Summary

FewShotSampler._init_fewshot_sampling_sequential rotates the list returned by task.fewshot_docs() in place. That list is the task's memoized _fewshot_docs, returned by reference, so the rotation mutates shared state. When an evaluation runs several few-shot seeds against the same task (variance estimation, i.e. few_shot_iterations > 1), each seed rotates the already-rotated pool, so the offsets accumulate and every seed after the first selects the wrong examples. Already-built cache entries are corrupted too, because self._fewshot_cache[variance_seed] stores a reference to the same shared list instead of a copy, so all seeds end up aliasing one over-rotated list.

The sibling _init_fewshot_sampling_random already copies the pool with list(...) before touching it; the sequential path does not.

Only the sequential selection method is affected. random copies the pool, and balanced builds its own per-label lists and never mutates the pool.

Reproduction

from lighteval.tasks.lighteval_task import LightevalTask, LightevalTaskConfig
from lighteval.tasks.prompt_manager import FewShotSampler
from lighteval.tasks.requests import Doc

config = LightevalTaskConfig(
    name="demo", prompt_function=lambda _, __: None, hf_repo="", hf_subset="default",
    metrics=[], few_shots_split="test", few_shots_select="sequential",
)
task = LightevalTask(config)
task._fewshot_docs = [Doc(query=str(i), choices=["A", "B"], gold_index=0) for i in range(10)]
original = [d.query for d in task._fewshot_docs]

sampler = FewShotSampler(task)
# Variance evaluation scores the same item with several few-shot seeds.
for seed in (0, 1, 2):
    picked = [d.query for d in sampler.sample_fewshot_examples(num_fewshot=2, variance_seed=seed)]
    rot = (2 * seed) % len(original)
    expected = (original[rot:] + original[:rot])[:2]
    print(f"seed {seed}: picked={picked}  expected={expected}")

print("pool after sampling:", [d.query for d in task.fewshot_docs()])
print("pool unchanged:", [d.query for d in task.fewshot_docs()] == original)

Output on current main:

seed 0: picked=['0', '1']  expected=['0', '1']
seed 1: picked=['2', '3']  expected=['2', '3']
seed 2: picked=['6', '7']  expected=['4', '5']
pool after sampling: ['6', '7', '8', '9', '0', '1', '2', '3', '4', '5']
pool unchanged: False

Seed 2 should select the pool rotated by 4 (['4', '5']) but gets a rotation of 6 (['6', '7']), because seed 1 already rotated the shared pool by 2 and seed 2 rotates that result by 4 more. The pool itself is left rotated after the run.

Root cause

In src/lighteval/tasks/prompt_manager.py:

def _init_fewshot_sampling_sequential(self, num_fewshot, variance_seed):
    fewshotpool = self.task.fewshot_docs()          # memoized list, returned by reference
    for _ in range(num_fewshot * variance_seed):
        fewshotpool.append(fewshotpool.pop(0))      # rotates the shared pool in place
    self._fewshot_cache[variance_seed] = fewshotpool

fewshot_docs() returns self._fewshot_docs, so the in-place rotation persists across calls and every cache entry aliases the same list.

Expected behavior

Sequential selection for seed s should return the pool rotated by num_fewshot * s from the original order, independently per seed, and leave task.fewshot_docs() unchanged.

Fix

Copy the pool before rotating, matching _init_fewshot_sampling_random:

fewshotpool = list(self.task.fewshot_docs())

I have a patch plus a regression test ready and will open a PR.