Add Edge Probing Suite to Full Benchmark

Assessment

Research direction

Description

Getting set up

First steps

1. Read the whole issue, then the project's contributing guide.
2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
3. Fork the repository and make your change on a branch.
4. Open a pull request that references the issue number.

More from bigscience-workshop/evaluation

• Add MKQA to Full Benchmark

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 25/100

  bigscience-workshop/evaluation#73 · Opened Feb 7, 2022

• Refactor task template to merge multilingual.json and english.json

  May be free again @wilsonyhlee claimed this 1856 days ago, and no pull request is open. Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  bigscience-workshop/evaluation#64 · 1 assignee · Opened Sep 2, 2021

• Wrap evaluation benchmark using HF-trainer

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 5/5 Over a week Newbie friendliness 20/100

  bigscience-workshop/evaluation#61 · 2 comments · 3 reactions · Opened Aug 25, 2021

• Setup testing

  May be free again @tttyuntian claimed this 1872 days ago, and no pull request is open. Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel
  engineering

  bigscience-workshop/evaluation#57 · 2 comments · 3 assignees · Opened Aug 19, 2021

• Start overleaf for benchmark tech report

  May be free again @arunraja-hub claimed this 1867 days ago, and no pull request is open. Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel
  documentation

  bigscience-workshop/evaluation#54 · 1 comment · 3 reactions · 2 assignees · Opened Aug 16, 2021

All issues in bigscience-workshop/evaluation

Similar issues

• Layers of type "3D Unknown" and "3D Measured Unknown" are refused, so FileGDB multipatch layers can't be read

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 2/5 1-3 hours Newbie friendliness 78/100

  geopandas/pyogrio#702 · Opened Oct 4, 2026

• [Bug]: flashinfer_moe_ep_mega_deep_gemm: 'ImportError: generic_type: type "Runtime" is already registered!' with vendored DeepGEMM

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel
  quantization

  Difficulty 2/5 1-3 hours Newbie friendliness 78/100

  vllm-project/vllm#59904 · Opened Oct 4, 2026

  Maintainers usually reply within 1 day

• [Paheal] Site change broke image URL extraction when using metadata flag

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 2/5 1-3 hours Newbie friendliness 72/100

  mikf/gallery-dl#9791 · Opened Oct 3, 2026

• Speaker custom-fields CSV export returns 404 instead of downloading data

  Possibly taken @techvoyager-varun claimed this today. Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 2/5 1-3 hours Newbie friendliness 88/100

  fossasia/eventyay#6151 · 1 comment · Opened Oct 3, 2026

  Maintainers usually reply within 1 day

• The stages run twice on 59 of 2,082 out-of-corpus wordings (2.8%); the roadmap quotes the corpus figure alone

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel
  P4: low tooling

  Difficulty 2/5 1-3 hours Newbie friendliness 88/100

  jeffknupp/association#318 · Opened Oct 3, 2026

More Python issues