huggingface/lighteval

[FT] showing count in Markdown summary table

開放

#804 建立於 2025年6月9日

 (5 則留言) (1 個反應) (0 位負責人)Python (514 個分叉)auto 404
featuregood first issue

倉庫指標

星標
 (2,496 顆星)
PR 合併指標
 (PR 指標待抓取)

描述

Issue encountered

I am seeing this output from lighteval endpoint litellm with lighteval==0.10.0:

|          Task          |Version|    Metric     |Value|   |Stderr|
|------------------------|------:|---------------|----:|---|-----:|
|all                     |       |ether0_accuracy|    0|±  |     0|
|community:ether0:loose:0|      0|ether0_accuracy|    0|±  |     0|

At a glance, it's unclear if I have misconfigured LightevalTaskConfig.hf_filter, or if accuracy truly was 0%.

(Note: in actuality, there were 10 questions, all incorrect.)

Solution/Feature

Showing a count of questions run, to clarify the size of the evaluation.

Possible alternatives

None

貢獻者指南