Autoloop PR/issue should report cumulative performance improvement
評価
この issue はまだ評価されていません。
説明
Problem
Autoloop programs that optimize a metric over many iterations don't report the cumulative improvement anywhere visible — not in the PR body, not in the issue, not in the state file summary. A reader of tsessebe PR #297 or issue #189 can see the current best metric but has no idea where the program started or how much progress has been made.
This was observed on the tsb-perf-evolve program (41 iterations, fitness ratio optimization). The PR body says "Current best metric: 21.048" but never mentions the starting value or total improvement.
Root causes
Three gaps in autoloop.md:
1. No Initial Metric in Machine State
The state file schema tracks Best Metric but not the baseline metric from the first accepted iteration. Without this, there's no reference point to compute cumulative improvement.
Fix: Add an Initial Metric field to the Machine State table schema. Set it on the first accepted iteration and never overwrite it.
2. PR body template doesn't include cumulative improvement
Step 5c says to update the PR body with "the latest metric and a summary of the most recent accepted iteration" but never specifies showing start-to-finish improvement.
Fix: Update the Step 5c PR body template to include: "Fitness: {best_metric} (started at {initial_metric}, {improvement_pct}% improvement)" or similar.
3. pending-ci fitness values never flow back into PR/issue
When the sandbox can't run the evaluation command (e.g., bun not available), iterations are pushed as pending-ci. After CI runs the benchmark and produces a fitness number, that result never flows back into the PR body, issue status comment, or state file. The PR stays stuck showing the last sandbox-measured metric.
Fix: Either (a) add a post-CI callback step that updates the PR body/state file with the CI-measured fitness, or (b) at minimum, have the next iteration read CI results from the previous run and update the state file retroactively before proposing a new change.
Expected behavior
A reader of any Autoloop PR or program issue should be able to see at a glance:
- Where the metric started (initial/baseline value)
- Where it is now (current best)
- Total improvement (absolute delta and percentage)
- A brief improvement trajectory (e.g., in the PR body or status comment)
References
- Observed on: githubnext/tsessebe PR #297, issue #189
- State file:
tsb-perf-evolve.md - Workflow definition:
.github/workflows/autoloop.md
- 主要言語
- Python
- スター
- 72
- フォーク
- 6
- PR マージ指標
- 30日以内にマージされた PR はありません
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
githubnext/autoloop のほかの issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
githubnext/autoloop#71 ·
-
難易度 2/5 1〜3時間 初心者へのやさしさ 85/100
githubnext/autoloop#54 ·
-
Releases オープン
難易度 4/5 3〜5日 初心者へのやさしさ 35/100
githubnext/autoloop#72 · コメント 1 件 ·
-
githubnext/autoloop#65 · リアクション 1 件 · 担当者 2 名 ·
-
githubnext/autoloop#63 · リアクション 1 件 · 担当者 2 名 ·
githubnext/autoloop の issue をすべて見る
似ている issue
-
triage/confirmed
難易度 2/5 1〜3時間 初心者へのやさしさ 88/100
agentscope-ai/agentscope#2775 ·
-
comp/desktop P3 type/bug
難易度 1/5 1時間未満 初心者へのやさしさ 92/100
NousResearch/hermes-agent#118866 ·
-
bug
難易度 1/5 1時間未満 初心者へのやさしさ 90/100
apache/cloudstack#14222 ·
-
難易度 2/5 1〜3時間 初心者へのやさしさ 76/100
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100