Roadmap: making dit the best form-detection and classification tool

Abierto
  • ¿Algo no está bien?
#79 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
8/100
Tipo de issue
Nueva funcionalidad
Claridad
Necesita aclaración
Estado de actividad
Activo
Stack tecnológico
go
Área
cli, machine-learning, security

Línea de trabajo

This is a multi-phase roadmap, not a single task, so do not start here. Read the Phases checklist and the linked child issues (#3, #4, #5 for CLI work; #10, #11, #13 for detections), then pick one concrete child issue. Relevant code lives in crf/, classifier/ and captcha/. Done means that child issue's acceptance criteria are met, not this roadmap's checkboxes.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Where we are

dît is a Go library + CLI that classifies page types (login, error, landing, ...), form types (login, registration, search, password recovery, contact, order, ...) and individual field types (username, password, email, ...) with an in-repo CRF implementation (crf/: forward-backward, Viterbi, trainer) and linear classifiers (classifier/), with zero external ML dependencies. It ships train/evaluate/cross-validation tooling, a dit-collect data collector, a pretrained model auto-downloaded from Hugging Face (~93 MB), and an early captcha/ package. Honest caveats: the model is large for a CLI tool, there's no published accuracy comparison against prior art, CLI ergonomics are still minimal (no stdin, no batch, single output format), and three open PRs (#75, #76, #77) literally titled "PR title" need triage.

Landscape

Project Strengths Link
Formasaurus The direct prior art: Python form/field classifier (LogReg form types + CRF field types), trained on 1000+ annotated forms — but Python-only, heavy sklearn/sklearn-crfsuite deps, and effectively dormant https://github.com/scrapinghub/Formasaurus
urlscan.io Renders pages in a real browser, records DOM/network/screenshot, community verdicts — but SaaS, no structural form/field classification, rate-limited API https://urlscan.io
PhishTank Crowd-sourced phishing URL feed — reactive blocklist, zero structural analysis of unseen pages https://phishtank.org
Playwright/headless scrapers Can find forms on JS-rendered pages, but classify nothing; users hand-roll selectors https://playwright.dev

Formasaurus proved the approach (docs, PyPI) and then stalled. Nobody offers a single-binary, dependency-free form/field/page classifier usable in security pipelines — that is exactly dît's lane.

What "best" means

  1. Beat the prior art, provably: publish cross-validated accuracy vs Formasaurus on a shared/annotated corpus (target: ≥ its reported form-type accuracy, with per-class F1 in the README).
  2. Pipeline-native: cat page.html | dit run - and dit run -l urls.txt --json work, so dît composes with httpx/katana/nuclei-style toolchains (#3, #4, #5).
  3. Fast and self-contained: classify a page in <100 ms on commodity hardware after one-time model download, with no Python, no network.
  4. Detections nobody else does structurally: honeypot fields (#10), CAPTCHA presence/type (#11), and phishing-form signals (#13) as first-class outputs, benchmarked against labeled sets rather than blocklists.
  5. Model you can retrain in an afternoon: documented dit train path with custom labels (#9) and a model ≤ 25 MB (quantized/pruned) as the default download.

Phases

Phase 1 — CLI & pipeline ergonomics
  • Read HTML from stdin (#3)
  • Batch mode for multiple URLs (#4)
  • Multiple output formats: table, csv, json (#5)
  • pre-commit config (#26) and triage/rename placeholder PRs #75, #76, #77
Phase 2 — Security detections (the differentiator)
  • Honeypot field detection (#10)
  • CAPTCHA detection (#11) — build on the existing captcha/ package
  • Phishing form detection (#13)
  • Form change monitoring (#12) — turns dît from a classifier into a monitoring primitive
Phase 3 — Reach & model quality
  • Server mode with HTTP API (#6)
  • JavaScript rendering for SPA forms (#7)
  • Multi-language form support (#8) — Formasaurus is English-biased; this is an open gap
  • Custom label support for training (#9)
  • Published benchmark vs Formasaurus + model-size reduction (claims 1 and 5)

Open questions

  • Can the 93 MB model.json be quantized/pruned (or split per-task: page/form/field) without losing accuracy, so go install users aren't surprised by the download?
  • What is the licensing/provenance story for the Hugging Face training data — can we accept community-contributed annotations?
  • Should JS rendering (#7) live in core (chromedp dependency) or as a separate dit-render companion to keep the zero-dep promise?

🤖 Filed by Claude Code

Generated by Claude Code

Lenguaje dominante
Go
Estrellas
19
Forks
4
Merge medio
15 min
PR fusionados (30 d)
2

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de HappyHackingSpace/dit

Todos los issues de HappyHackingSpace/dit

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.