ShishirPatil/gorilla

[Apibench] Resume interrupted LLM generations from last generation

开放

#484 创建于 2024年6月24日

 (1 条评论) (0 个反应) (0 位负责人)Python (1,399 个派生)batch import
apibench-datagood first issue

仓库指标

星标
 (12,997 个星标)
PR 合并指标
 (30 天内没有已合并 PR)

描述

Describe the issue When we try to launch LLM generations on APIBench, if the generation is interrupted midway, currently the partial file is deleted and restart generation from start. What we should instead do is check where the generation stopped and resume from the last generated index. This is specially critical for large models where generation can take >1 hours.

ID datapoint 2. Provider: This affects all 3: TorchHub/HuggingFace/PyTorch Hub

File that needs to be edited: https://github.com/ShishirPatil/gorilla/blob/main/eval/get_llm_responses.py

贡献者指南