Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Proposal: correctness benchmark + reviewed skill files for AI assistants (outside core)

Abierto
  • ¿Algo no está bien?
#14,428 5 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
machine-learning, markdown, python
Área
developer-experience, documentation, testing

Línea de trabajo

Start by reviewing AGENTS.md and the linked discussions (#13288, #13592, and #13702), then define the experimental pilot outside the core mne module. The proposed pilot is 20 benchmark questions with explicit pass/fail criteria, one reviewed skill file, and a rerunnable comparison script; the issue also needs decisions on repository placement, reviewer ownership, and scope before implementation.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Hi all,

AI assistants and MNE keep coming up, both here and on the forum. I went back through the earlier discussions and wanted to suggest something concrete that I think addresses the concerns raised there.

What's been discussed so far

  • #13288: the Steering Council decided not to promote an AI-driven MCP server, mainly because we can't vouch for the correctness of what these tools produce.
  • #13592: still-open question about an MCP server for MNE.
  • #13702: the OSA chat widget PR. @cbrnr's BDF status channel example showed a confidently wrong answer, Kapa came up as an alternative, and @wmvanvliet raised the point about not putting MNE branding on something we haven't vetted.
  • The forum thread (https://mne.discourse.group/t/ai-agents-to-work-with-mne/11744): Maximilien suggested starting with .md skills/rules and asked for volunteers, and Daniel Borek pointed to how PyMC and marginaleffects did this in separate repos. Community skills are already appearing too, e.g. ClaudeEEG (announced on the forum in April), which bundles a copy of our tutorials, but as far as I can tell none of these have been reviewed or measured for correctness.
  • On the developer side, AGENTS.md is already in place and was recently refined (#14414), so this would be the user-facing counterpart.

The common thread for me is that nobody can currently say how often these tools are wrong for MNE questions. Our users are already using ChatGPT, Claude, Copilot etc. whether we recommend them or not, so I think measuring that is more useful than adding another tool.

What I'd like to try

  1. A small benchmark. Real MNE questions (from forum threads and known pitfalls like the BDF status channel, filtering epochs vs. raw, ICA on high-passed data, decoding/CV leakage, choosing an inverse method), each with clear pass/fail criteria checked by someone who knows the topic. Where possible, questions would ask for code that has to run and give a checkable result, rather than free text. Plus a simple script so anyone can rerun it with whatever model they already use. No API cost to the project. This could also give us actual data for deciding on #13702.
  2. Reviewed skill files. Plain Markdown notes on MNE best practices and common mistakes, linking back to our tutorials, and only marked stable after someone experienced has reviewed them. The benchmark would show whether they actually help.
  3. Kept outside core. A separate experimental repo, either under mne-tools if people are comfortable with that, or under my account to start with. No new module, no dependency, nothing in mne itself, and no MNE branding on a chat bot.

How I'd go about it

I'd start small, with 20 questions, one draft skill file, and a comparison of 2–3 assistants (with and without existing skills like ClaudeEEG) graded by hand, and post the results here. If that's useful, I'd expand it into a fuller set. I'm happy to do this as volunteer work. If it turns out well and someone is interested in mentoring, it might also make a reasonable GSoC 2027 idea.

Where I'd appreciate your input

  • Does this direction sit OK with the Steering Council's earlier position, or are there concerns I'm missing?
  • mne-tools repo or personal repo to start?
  • If you have favourite "everyone gets this wrong" questions from the forum, I'd love to collect them for the pilot.

Thanks, and no rush on this. I know everyone's bandwidth is limited, so I'd rather get the scope right before starting.

Lenguaje dominante
Python
Estrellas
3.6k
Forks
1.6k
Merge medio
1 d 4 h
PR fusionados (30 d)
73

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de mne-tools/mne-python

Todos los issues de mne-tools/mne-python

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.