Start overleaf for benchmark tech report

Assessment

This issue has not been assessed yet.

Description

Getting set up

First steps

1. Read the whole issue, then the project's contributing guide.
2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
3. Fork the repository and make your change on a branch.
4. Open a pull request that references the issue number.

More from bigscience-workshop/evaluation

• Add MKQA to Full Benchmark

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 25/100

  bigscience-workshop/evaluation#73 · Opened Feb 7, 2022

• Refactor task template to merge multilingual.json and english.json

  May be free again @wilsonyhlee claimed this 1857 days ago, and no pull request is open. Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  bigscience-workshop/evaluation#64 · 1 assignee · Opened Sep 2, 2021

• Wrap evaluation benchmark using HF-trainer

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 5/5 Over a week Newbie friendliness 20/100

  bigscience-workshop/evaluation#61 · 2 comments · 3 reactions · Opened Aug 25, 2021

• Setup testing

  May be free again @tttyuntian claimed this 1873 days ago, and no pull request is open. Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel
  engineering

  bigscience-workshop/evaluation#57 · 2 comments · 3 assignees · Opened Aug 19, 2021

• translate validation prompts into all training languages

  May be free again @epavlick claimed this 1879 days ago, and no pull request is open. Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel
  multilingual simple_benchmark

  bigscience-workshop/evaluation#50 · 9 comments · 2 assignees · Opened Aug 12, 2021

All issues in bigscience-workshop/evaluation

Similar issues

• Action calls retired claude-3-5-haiku-20241022, generating failing API requests for every user

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 1/5 Under an hour Newbie friendliness 91/100

  anthropics/claude-code-security-review#154 · Opened Oct 4, 2026

• Mailer.send() raises NameError: add_charset, charset_QP, Header and Encoders are not imported (1.20260805.0)

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 1/5 1-3 hours Newbie friendliness 92/100

  web2py/py4web#1069 · Opened Oct 4, 2026

• feat: offline grade-run must discover entry-nested harness bundles

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel
  enhancement P2

  Difficulty 2/5 1-3 hours Newbie friendliness 78/100

  Toloka/tolokaforge#1776 · Opened Oct 4, 2026

  Maintainers usually reply within 1 day

• [bug] 本地会话日志回退在事件循环上同步读取 sessions/*.jsonl，长会话下阻塞同 loop 请求

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 2/5 1-3 hours Newbie friendliness 88/100

  TencentCloud/Octop#1622 · Opened Oct 4, 2026

  Maintainers usually reply within 1 day

• [arch] fleet events list: emitted_at sorts lexically, misordering mixed-format journals

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel
  arch area:fleet priority:p3 severity:low track:hosted-product

  Difficulty 2/5 1-3 hours Newbie friendliness 85/100

  ntindle/spark-vm#1010 · Opened Oct 4, 2026

  Maintainers usually reply within 1 day

More Python issues