Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Proposal: correctness benchmark + reviewed skill files for AI assistants (outside core)

Aperta
  • Qualcosa non va?
#14,428 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
machine-learning, markdown, python
Ambito
developer-experience, documentation, testing

Direzione di ricerca

Start by reviewing AGENTS.md and the linked discussions (#13288, #13592, and #13702), then define the experimental pilot outside the core mne module. The proposed pilot is 20 benchmark questions with explicit pass/fail criteria, one reviewed skill file, and a rerunnable comparison script; the issue also needs decisions on repository placement, reviewer ownership, and scope before implementation.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Hi all,

AI assistants and MNE keep coming up, both here and on the forum. I went back through the earlier discussions and wanted to suggest something concrete that I think addresses the concerns raised there.

What's been discussed so far

  • #13288: the Steering Council decided not to promote an AI-driven MCP server, mainly because we can't vouch for the correctness of what these tools produce.
  • #13592: still-open question about an MCP server for MNE.
  • #13702: the OSA chat widget PR. @cbrnr's BDF status channel example showed a confidently wrong answer, Kapa came up as an alternative, and @wmvanvliet raised the point about not putting MNE branding on something we haven't vetted.
  • The forum thread (https://mne.discourse.group/t/ai-agents-to-work-with-mne/11744): Maximilien suggested starting with .md skills/rules and asked for volunteers, and Daniel Borek pointed to how PyMC and marginaleffects did this in separate repos. Community skills are already appearing too, e.g. ClaudeEEG (announced on the forum in April), which bundles a copy of our tutorials, but as far as I can tell none of these have been reviewed or measured for correctness.
  • On the developer side, AGENTS.md is already in place and was recently refined (#14414), so this would be the user-facing counterpart.

The common thread for me is that nobody can currently say how often these tools are wrong for MNE questions. Our users are already using ChatGPT, Claude, Copilot etc. whether we recommend them or not, so I think measuring that is more useful than adding another tool.

What I'd like to try

  1. A small benchmark. Real MNE questions (from forum threads and known pitfalls like the BDF status channel, filtering epochs vs. raw, ICA on high-passed data, decoding/CV leakage, choosing an inverse method), each with clear pass/fail criteria checked by someone who knows the topic. Where possible, questions would ask for code that has to run and give a checkable result, rather than free text. Plus a simple script so anyone can rerun it with whatever model they already use. No API cost to the project. This could also give us actual data for deciding on #13702.
  2. Reviewed skill files. Plain Markdown notes on MNE best practices and common mistakes, linking back to our tutorials, and only marked stable after someone experienced has reviewed them. The benchmark would show whether they actually help.
  3. Kept outside core. A separate experimental repo, either under mne-tools if people are comfortable with that, or under my account to start with. No new module, no dependency, nothing in mne itself, and no MNE branding on a chat bot.

How I'd go about it

I'd start small, with 20 questions, one draft skill file, and a comparison of 2–3 assistants (with and without existing skills like ClaudeEEG) graded by hand, and post the results here. If that's useful, I'd expand it into a fuller set. I'm happy to do this as volunteer work. If it turns out well and someone is interested in mentoring, it might also make a reasonable GSoC 2027 idea.

Where I'd appreciate your input

  • Does this direction sit OK with the Steering Council's earlier position, or are there concerns I'm missing?
  • mne-tools repo or personal repo to start?
  • If you have favourite "everyone gets this wrong" questions from the forum, I'd love to collect them for the pilot.

Thanks, and no rush on this. I know everyone's bandwidth is limited, so I'd rather get the scope right before starting.

Lingua principale
Python
Stelle
3.5k
Fork
1.6k
Merge medio
1g 2h
PR unite (30g)
72

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di mne-tools/mne-python

Tutte le issue di mne-tools/mne-python

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.