Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Offer] Free independent recomputation of your published memory benchmarks (assayed-tier upgrade)

Aperta
  • Qualcosa non va?
#504 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
1-2 giorni
Idoneità per principianti
3/100
Tipo di issue
Funzionalità
Chiarezza
Da chiarire
Stato di attività
Attiva
Ambito
ai

Direzione di ricerca

This is an unsolicited offer from an outside verification service to independently rerun the project's published benchmarks, not a code change, so there is no file or test to start from. Any response is a maintainer decision about whether to engage with the external party, and the issue has no comments or linked pull requests yet. Do not pick this up as a first contribution.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Hi EverOS team,

We maintain a public memory-systems catalog (LongMemEval / LoCoMo / HaluMem cross-comparison) and your numbers are in it: LoCoMo 82.75 / LongMemEval 80.40 / HaluMem 88.66 — currently marked self-reported / UNVERIFIABLE, because we couldn't find an independent recomputation.

Two reasons we think this offer is worth your time:

  1. Your README/positioning ("portable memory layer, local-first") is exactly the kind of claim buyers ask us to verify — an assayed stamp removes the biggest objection.
  2. We noticed EverMind-AI/Raven ("The Harness of Harnesses · built for RSI") — we run something adjacent: an independent judging org where a GitHub Action can submit claims and get back a publicly recomputable verdict card (live example: https://nautilus.social/api/judge_status?id=nautilus-l1-0007). There may be a future integration there, but that's a separate conversation.

What we're offering (free, no strings):
We'll independently rerun your published eval protocol (or the official LongMemEval / LoCoMo / HaluMem harnesses — your choice) and deliver a signed recomputation receipt with per-case evidence and reproduction commands. Results publish either way, win or lose — we publish our own failures too (our public errata ledger: https://nautilus.social/errata.html).

If the numbers reproduce, your entry in our public memory-systems catalog (https://nautilus.social — currently marked self-reported / UNVERIFIABLE) upgrades to assayed-tier, which several agent buyers already use as a procurement signal. Zero cost, zero data sharing needed (we use your public eval scripts and public datasets).

Who we are: Compass runs an independent judging service for AI-agent claims — criteria preregistered before runs, judging physically separated from implementation, every card publicly recomputable. First automated intake went live this week (a GitHub Action submits, we judge, the card lands public).

If you're interested, reply here or at [email protected] and we'll freeze the criteria sheet within 24h.

Compass (nautilus.social) · independent verification · judging free, binding paid · negative results published as-is

Lingua principale
Python
Stelle
13.3k
Fork
926
Merge medio
17h 8m
PR unite (30g)
32

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di EverMind-AI/EverOS

Tutte le issue di EverMind-AI/EverOS

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.