Full-database audit — findings, verification method, and fix branches

Abierto
  • ¿Algo no está bien?
#1,700 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
25/100
Tipo de issue
Error
Claridad
Necesita aclaración
Estado de actividad
Tranquilo
Área
data, tooling

Línea de trabajo

Empieza leyendo los informes de auditoría enlazados y las secciones metadat/ y cht/; después, acuerda con los maintainers una única rama centrada en un solo tema. Se considera terminado cuando se haya enviado como PR independiente un fix de alcance reducido y verificado de forma independiente; la evaluación más amplia y las decisiones sobre la ortografía siguen sin resolverse.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

I ran a full audit of this repo: all of metadat/ (660 dats, ~727k game blocks) and cht/ (28,301 files, 44 systems). Everything below is condensed — the full reports, per-class verification verdicts and the scanning scripts are at https://github.com/ArjunKdaf/lrdb-audit (MIT — take anything useful). Posting this before I start sending fixes, so the volume doesn't come as a surprise and you can tell me where to start.

Method. Fresh clone of master (6fd53f9). Every dat parsed with a quote-aware clrmamepro parser — naive paren-counting silently corrupts on titles like Odekake Lester - Lelele no Le (^^; (Japan), worth knowing for any tooling on your side. Metadata checked against the union of no-intro/redump/tosec per platform. Cheat suspects were verified against the parsers of the cores that consume the files (Genesis Plus GX, PicoDrive, Gearsystem, Beetle PSX, PCSX-ReARMed, SwanStation, RetroArch's cheat_manager), cited to file and line at pinned commits, plus published code lists, Datel docs and this repo's git history — nothing is reported from pattern-matching alone. I used AI agents (Claude) for the legwork, same setup as my earlier PRs here; it's faster than doing it by hand and I have API budget to spare this month.

Wrong metadata — 10,191 orphaned CRCs, 4,423 stale titles, 1,140 duplicate CRCs database-wide:

  • ~7,800 of the orphans are already fixed on my fork: machine-verified resyncs of N64, NES, Mega Drive, GBA, SNES, GB, SMS, GBC, Game Gear and NDS, same method as #1691. Held back deliberately — they'll come as single-platform PRs at whatever pace works for review.
  • Cheapest remaining win: Sega 32X — 840 stale titles, 0 dead CRCs, a pure rename pass. Then Famicom Disk System, PC Engine, Jaguar, SG-1000 (700–1,200 findings each).
  • ~4,200 blocks are orphaned by filename alone: N64 "(ByteSwapped)"/"(BigEndian)" esrb/franchise files, "Magnavox - Odyssey 2"/"Microsoft - MSX 2" space variants coexisting with canonical spellings.
  • Small mechanical fixes: one corrupt dat (lost-level-archive Arduboy — a doubled quote, 6 of 29 games unparseable, one-character fix), 4 invalid hashes, 180 dats missing the final trailing newline, 1,202 byte-identical duplicate blocks, ~230 name fields with invisible NBSP/trailing-whitespace.
  • One gap in my own open PR: #1691 doesn't cover NGPC releasemonth and serial (89 findings). Fold into the branch or follow-up — your call.

Missing metadata:

  • PS1/PS2/PSP: ~33k metadata blocks are serial-keyed with no crc fields, so none of it joins into the RDBs — this is why PSP shows zero metadata despite 7 populated dats. A serial→crc backfill (or serial-aware join in the build tooling) lights up three platforms at once.
  • 69 of 122 platforms have no metadata dats at all. Flagging the scale only — that's authoring work, not fixing.

Cheats — 1,462 confirmed defects, every class verified against core source:

  • Actively harmful, worth fixing first: 2 Ristar codes under Game Gear that Gearsystem misroutes into unvalidated emulated-RAM writes, plus 3 misfiled Genesis codes whose correct copies already exist in the Genesis file.
  • 848 PSX "Game Buster" codes fragmented into 4-char +-pieces — dead in all three PSX cores. Mechanical re-join into canonical 8+4 form verified safe for all 848.
  • 924 Game Gear Game Genie codes with + where cores require - — dead in GPGX/PicoDrive. The codes themselves are genuine; fix is mechanical.
  • 179 individually bad codes (letter-O-for-zero batches, illegal Game Genie letters, float-mangled values like 8095000000.00, prose in code fields), plus 89 N64 codes hit by one systematic value/address rotation bug — reconstruction verified line-by-line against gamehacking.org.
  • Systemic: all 752 fMSX files share one conversion bug (conflicting duplicate key); 3 files are wholly Mednafen-format so RetroArch loads zero cheats from them; 406 unparseable lines; 59 files declare wrong cheat counts.
  • ~30,600 blank-code entries trace to the Feb 2020 bulk import (7cd0d683 and siblings — #994 already noted the gamehacking.org translator left entries blank). Restorable from the source. The separate 24k desc-only NDS entries are intentional folder headers, untouched.
  • Deliberately not touched: 2 odd-looking but legitimate published XplorerPro codes, 8 codes unverifiable against any source, and 1 defect that exists verbatim at its published source (upstream of this repo).

Your call before anyone writes a branch: the retired KA ESRB rating on 1,322 entries (keep as historical or migrate?); 264 developer/publisher spelling clusters (Koei/KOEI at 254-vs-79 is the largest — normalize or leave?).

Core-side note: smsplus-gx has an empty retro_cheat_set — cheats never apply in that core regardless of data quality. Filing that upstream separately.

Plan. Single-topic fix branches, one PR at a time, at your review pace. If you'd rather I start somewhere specific, say so.

Lenguaje dominante
Python
Estrellas
1.5k
Forks
889
Merge medio
2 d 10 h
PR fusionados (30 d)
24

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de libretro/libretro-database

Todos los issues de libretro/libretro-database

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.