huggingface/lighteval

[FT] showing count in Markdown summary table

オープン

#804 opened on 2025/06/09

 (5 件のコメント) (1 件のリアクション) (0 人の担当者)Python (514 件のフォーク)auto 404
featuregood first issue

Repository metrics

Stars
 (2,496 個のスター)
PR merge metrics
 (PR metrics pending)

説明

Issue encountered

I am seeing this output from lighteval endpoint litellm with lighteval==0.10.0:

|          Task          |Version|    Metric     |Value|   |Stderr|
|------------------------|------:|---------------|----:|---|-----:|
|all                     |       |ether0_accuracy|    0|±  |     0|
|community:ether0:loose:0|      0|ether0_accuracy|    0|±  |     0|

At a glance, it's unclear if I have misconfigured LightevalTaskConfig.hf_filter, or if accuracy truly was 0%.

(Note: in actuality, there were 10 questions, all incorrect.)

Solution/Feature

Showing a count of questions run, to clarify the size of the evaluation.

Possible alternatives

None

コントリビューターガイド