Serving benchmark: cold fit, warm predict, many tenants

未关闭
  • 哪里不对？
#104 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
35/100
Issue 类型
功能
描述清晰度
描述清楚
活跃度
活跃
技术栈
python
领域
machine-learning, performance

调研方向

Start by reading bench_workload.py, sweep_admission.py, and bench_concurrency.py, then inspect benchmarks/bench_vs_hosted_api.py for the hosted baseline. Build one script that runs the full suite and writes JSON, measuring cold fit, warm predict latency, and multi-tenant throughput and tail latency, including eviction and disk restores. Done means reproducible results for at least L4 and A40 are published in docs/benchmarks.md with exact commands.

由索引模型根据 Issue 内容生成。

描述

area/latency enhancement help wanted P2 roadmap

Publish a serving benchmark for tabular foundation models. The existing comparisons are accuracy leaderboards. Nobody publishes what it costs to serve these models.

Measure

  • Cold fit: time by rows × features.
  • Warm predict: p50 and p99 latency against a cached context, by batch size.
  • Many tenants: throughput and tail latency with N tenants and M tables on one GPU, including eviction and restores from disk.
  • Baselines: the model in your own process, and the hosted API where the terms allow it (benchmarks/bench_vs_hosted_api.py).

Deliver

  • One script that runs the whole suite and writes JSON, reusing bench_workload.py, sweep_admission.py and bench_concurrency.py.
  • Results on at least L4 and A40 in docs/benchmarks.md, with the exact commands to reproduce them.
主要语言
Python
星标
1
派生
1
平均合并
6 小时 45 分钟
30 天内合并 PR
66

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

VectorInstitute/tabctx 的其他 Issue

查看 VectorInstitute/tabctx 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。