huggingface/lighteval

[FT] Build in a way to specify specific IDs/Lines in Dataset to use as few-shot examples in the same split

开放

#634 创建于 2025年3月21日

 (3 条评论) (0 个反应) (0 位负责人)Python (514 个派生)auto 404
featuregood first issuehelp wanted

仓库指标

星标
 (2,496 个星标)
PR 合并指标
 (PR 指标待抓取)

描述

Solution/Feature

When creating a custom task, the few-shot samples can be specified via a split from the dataset. However, it would be convenient if there were a way to make it so I can specify samples from the split that will be used as the examples, similar to what LM Evaluation Harness does here in its yaml config:

test_split: train
fewshot_split: train
fewshot_config:
  sampler: id_sampler
  sampler_config:
    id_list: ["2022_21", "2022_88", "2022_143"]
    id_column: id
    exclude_from_task: true

Something along the lines of:

@dataclass
class LightevalTaskConfig:
    """Stored configuration of a given [`LightevalTask`].

    Arguments:
    ...
        hf_subset (str): Subset used for the current task, will be default if none is selected.
        hf_avail_splits (list[str]): All the available splits in the evaluation dataset
        evaluation_splits (list[str]): List of the splits actually used for this evaluation
        few_shots_split (str): Name of the split from which to sample few-shot examples
        few_shots_id_column (str): Column name in the dataset to use for the few_shot_id_list
        few_shots_id_list (list[str]): List of unique identifiers corresponding to the samples in the dataset, based on matching these IDs with the values in few_shots_id_column.
        few_shots_select (str): Method with which to sample few-shot examples
    ...
    """

The new arguments would be few_shots_id_column and few_shots_id_list, and wrote a rough description of what they would look like.

贡献者指南