huggingface/lighteval

[BUG] Optimize tokenization

开放

#732 创建于 2025年5月15日

 (1 条评论) (0 个反应) (1 位负责人)Python (514 个派生)auto 404
buggood first issuehelp wantedscience-team

仓库指标

星标
 (2,496 个星标)
PR 合并指标
 (PR 指标待抓取)

描述

Describe the bug

For now tokenization is bing made in a for loop, making the whole process very expensive for large benchmarks.

To Reproduce

run any big benchmarks

Expected behavior

Use batch tokenize to speed it up.

Version info

Please provide your operating system, lighteval version or commit if you installed from main, and pip/conda environment if your problem concerns dependencies.

贡献者指南