huggingface/lighteval

[BUG] Optimize tokenization

開放

#732 建立於 2025年5月15日

 (1 則留言) (0 個反應) (1 位負責人)Python (514 個分叉)auto 404
buggood first issuehelp wantedscience-team

倉庫指標

星標
 (2,496 顆星)
PR 合併指標
 (PR 指標待抓取)

描述

Describe the bug

For now tokenization is bing made in a for loop, making the whole process very expensive for large benchmarks.

To Reproduce

run any big benchmarks

Expected behavior

Use batch tokenize to speed it up.

Version info

Please provide your operating system, lighteval version or commit if you installed from main, and pip/conda environment if your problem concerns dependencies.

貢獻者指南