huggingface/lighteval

[FT] Build in a way to specify specific IDs/Lines in Dataset to use as few-shot examples in the same split

Aperta

#634 aperta il 21 mar 2025

 (3 commenti) (0 reazioni) (0 assegnatari)Python (514 fork)auto 404
featuregood first issuehelp wanted

Metriche repository

Star
 (2496 stelle)
Metriche merge PR
 (Metriche PR in attesa)

Descrizione

Solution/Feature

When creating a custom task, the few-shot samples can be specified via a split from the dataset. However, it would be convenient if there were a way to make it so I can specify samples from the split that will be used as the examples, similar to what LM Evaluation Harness does here in its yaml config:

test_split: train
fewshot_split: train
fewshot_config:
  sampler: id_sampler
  sampler_config:
    id_list: ["2022_21", "2022_88", "2022_143"]
    id_column: id
    exclude_from_task: true

Something along the lines of:

@dataclass
class LightevalTaskConfig:
    """Stored configuration of a given [`LightevalTask`].

    Arguments:
    ...
        hf_subset (str): Subset used for the current task, will be default if none is selected.
        hf_avail_splits (list[str]): All the available splits in the evaluation dataset
        evaluation_splits (list[str]): List of the splits actually used for this evaluation
        few_shots_split (str): Name of the split from which to sample few-shot examples
        few_shots_id_column (str): Column name in the dataset to use for the few_shot_id_list
        few_shots_id_list (list[str]): List of unique identifiers corresponding to the samples in the dataset, based on matching these IDs with the values in few_shots_id_column.
        few_shots_select (str): Method with which to sample few-shot examples
    ...
    """

The new arguments would be few_shots_id_column and few_shots_id_list, and wrote a rough description of what they would look like.

Guida contributor