关于 EdgeBench 长时程实验中逐轮反馈与 Agent 行为轨迹的数据 Research question: per-run feedback and adaptation traces

Aperta
  • Qualcosa non va?
#29 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Documentazione
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
machine-learning

Direzione di ricerca

Start with the evaluation platform and the related discussion in #7, which the issue identifies as covering time-varying scores. Ask whether the original experiments retained per-run records linking feedback, agent actions or revisions, and later outcomes, and whether any are available for research. Done means providing a pointer to suitable public data or clarifying its availability; the proposed offline analysis is research work rather than a specified repository change.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

各位老师好，

最近在看 EdgeBench 和 SForge，比较感兴趣的是 Agent 在长时程任务中如何利用中间评估反馈调整后续行为。

我注意到 #7 已经讨论过随时间变化的评分数据，维护者也提供了评测平台。不过我想进一步了解的是，原始实验中是否保留了能够按运行和时间步骤对齐的记录，例如每次评估的反馈、Agent 随后的行动或策略调整，以及后续的评估结果。

我目前关注 AI evaluation、feedback reliability 和 agent learning。一个想验证的问题是：中间评分的改善，是否真的能预测后续任务表现的提升？如果存在误导性反馈，能否通过额外的独立证据区分有效进展与虚假的局部改善？

如果有相关记录，我想先基于已有实验数据做一个小规模离线分析，比较简单的分数阈值与证据条件化方法的表现，暂时不涉及重新训练大模型。

想请教一下，这类逐轮反馈与行为记录是否有公开版本，或者是否存在可以用于科研分析的部分数据？如果团队已经在研究类似问题，也很希望了解现有进展，避免重复工作。

谢谢！

郭佰鑫
微信：Max-GuoBX

Thanks for releasing EdgeBench and SForge.

I'm interested in how agents use intermediate evaluation feedback during long-horizon tasks, particularly when an apparent improvement in a local score does not translate into reliable downstream progress.

I saw #7 and the link to the evaluation platform, but I'm looking for a slightly different artifact: temporally aligned records of evaluator feedback, subsequent agent actions or revisions, and later evaluation outcomes from the same run.

Were any such per-run traces retained from the original experiments, and are they available for research use?

I'd like to start with a small offline study of when intermediate feedback is actually informative enough to guide further adaptation, comparing simple score-based rules against evidence-conditioned alternatives. I wouldn't need a new training run or additional compute to begin.

If this is already covered by a public artifact or ongoing study, a pointer would be very helpful.

Thanks,
Baixin Guo (Max)

Lingua principale
Python
Stelle
474
Fork
20
Merge medio
1g 7h
PR unite (30g)
2

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di ByteDance-Seed/EdgeBench

Tutte le issue di ByteDance-Seed/EdgeBench

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.