Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

Evaluation: Add latency summary data

未关闭
  • 哪里不对？
#1,253 0 条评论 0 个 reaction 已指派 1 人 在 GitHub 查看

维护者通常 2 天内回复

@Ayush8923 已经在做这个了。

开始于 2026年10月14日。

  • #1252 来自 @Ayush8923 —— 未关闭

评估

这个 Issue 还没有评估数据。

描述

Is your feature request related to a problem?
Evaluation runs provide cost information for OpenAI calls but lack latency data. This prevents us from identifying which configurations or prompts are slower, impacting our ability to optimize evaluations effectively.

Describe the solution you'd like
Add a latency summary to each evaluation run next to the cost, including:

  • Average time of OpenAI calls in milliseconds, plus the number of calls for each step:
    • response: generating the answer
    • embedding: similarity scoring (v1 runs)
    • judge: LLM-as-judge scoring (v2 runs)
  • One overall average across all successful calls.
  • Display this data on existing run endpoints for frontend integration.
  • Ensure runs complete even if latency calculation fails.
Original issue

Is your feature request related to a problem? Please describe.
Evaluation runs show how much the OpenAI calls cost, but not how long they took. When we compare configs or prompts, we can't tell whether one setup is slower than another, or which step of the evaluation is slow: generating answers, embeddings, or judging. Response time is one of the main reasons fast evaluations exist, but we don't report it anywhere.

Describe the solution you'd like
Add a latency summary to each evaluation run, next to the existing cost:

  • The average time of an OpenAI call in milliseconds, plus the number of calls, for each step:
    • response: generating the answer for each question
    • embedding: similarity scoring (v1 runs)
    • judge: LLM-as-judge scoring (v2 runs)
  • One overall average across all calls in the run.
  • Only successful calls count. Failed calls and questions blocked by a guardrail are left out.
  • Shown on the existing run endpoints, so the frontend can display it on the run card next to cost.
  • If working out latency fails, the run should still complete.

Describe alternatives you've considered

  • Storing latency inside the cost field. Rejected: a problem with one could wipe out the other. Keeping them separate keeps them independent.
  • Using the total batch duration for batch runs. Rejected: batch time is mostly OpenAI queue time, not call time, so it would be misleading. Batch runs will show no latency.
  • Adding p95 / max / per-question latency now. Deferred: we'll start with averages. The call counts and per-question timings are stored so these can be added later.
主要语言
Python
星标
18
派生
11
平均合并
4 天 20 小时
30 天内合并 PR
9

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

ProjectTech4DevAI/kaapi-backend 的其他 Issue

查看 ProjectTech4DevAI/kaapi-backend 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。