integer_compression_codec: stored size anchors are impossible for the documented Varint starter
维护者通常 2 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 54/100
- Issue 类型
- 缺陷
- 描述清晰度
- 基本清楚
- 活跃度
- 活跃
- 技术栈
- python
- 领域
- testing-qa
调研方向
Read baseline_results.json and scoring_scheme.md, then reproduce the reported starter run with the listed sforge build and judge_local.py commands. Compare the per-dataset encoded sizes with the stored anchors and trace how the scoring calculation uses them. Decide how to handle the scale change while preserving the stated throughput anchors, then verify the starter and top-file scores against the issue's reported ranges.
由索引模型根据 Issue 内容生成。
描述
Task
integer_compression_codec
Claim
The stored size anchors in baseline_results.json can't come from the documented baseline. scoring_scheme.md says the baseline is the Varint starter, and Varint writes at least 8 bits per integer, yet five of eight anchors are below 8 (5.13 on docids) and all eight are below the starter's real size. Each dataset is off by a different factor, 1.23 to 8.23.
Impact
Every score uses these anchors. With the starter's exact sizes, the starter's raw score goes from 0.24-0.25 to 0.83-0.88 (documented 1.0) and our top file from 73.78 to raw 167.38-167.51, which clips on the 0..100 scale, so a fix needs a scale decision.
Evidence and how to reproduce
On community f92857a3, after python -m sforge build --task integer_compression_codec, grade the starter with python3 tools/judge_local.py --task-dir edgebench/tasks/integer_compression_codec --image edgebench.judge.integer_compression_codec:d5de821d0a38 --empty --out result.json and read metrics.per_dataset[].encoded_bytes in the structured result inside log_tail: encoded_bytes * 8 / n is 38.98 on docids, against the stored 5.13, on every run.
Limits
Throughput anchors stay as they are (Known issue 2). All real submissions here come from one agent's two runs.
- 主要语言
- SMT
- 星标
- 477
- 派生
- 20
- 平均合并
- 3 天 6 小时
- 30 天内合并 PR
- 3
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
ByteDance-Seed/EdgeBench 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 73/100
ByteDance-Seed/EdgeBench#48 ·
维护者通常 2 天内回复
-
FFmpeg: public result hides whether failure occurred before correctness checks可能已有人在做 @loopx-agent 于 4 天前认领。 未关闭
难度 4/5 3-5 天 新手友好度 35/100
ByteDance-Seed/EdgeBench#57 ·
维护者通常 2 天内回复
-
难度 2/5 1-3 小时 新手友好度 35/100
ByteDance-Seed/EdgeBench#51 ·
维护者通常 2 天内回复
-
难度 4/5 3-5 天 新手友好度 48/100
ByteDance-Seed/EdgeBench#50 ·
维护者通常 2 天内回复
-
schemathesis_datagen_pipeline: hidden Phase 1 imports names and format keys that TODO.md never gives未关闭
难度 3/5 1-2 天 新手友好度 48/100
ByteDance-Seed/EdgeBench#49 ·
维护者通常 2 天内回复
查看 ByteDance-Seed/EdgeBench 的全部 Issue
相似的 Issue
-
ci javascript tests
难度 2/5 1-3 小时 新手友好度 78/100
jonathanong/no-mistakes#3258 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 82/100
nikic/PHP-Parser#1186 ·
-
难度 2/5 1-3 小时 新手友好度 66/100
wultra/powerauth-tests#980 ·
-
area: cpu-engines area: fuzzer good first issue
难度 2/5 1-3 小时 新手友好度 91/100
hffgzktbcc-blip/ounce-core#402 ·
维护者通常 1 天内回复
-
bug
难度 2/5 1-3 小时 新手友好度 65/100
swc-project/swc-ai-issues#432 ·