Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Text Analysis Tutorial: Evaluation & Error Analysis

Aperta
  • Qualcosa non va?
#256 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Documentazione
Chiarezza
Abbastanza chiara
Stato di attività
Ferma
Stack tecnologico
fastapi, huggingface, jupyter-notebook, pandas, python, pytorch, scikit-learn
Ambito
data, documentation, machine-learning, testing

Direzione di ricerca

Inizia con la struttura proposta di tutorials/t12-eval-error-analysis/, in particolare notebooks/, src/metrics.py, src/error_analysis.py, src/calibration.py, data/README.md e tests/. Segui i requisiti di configurazione e quelli passo per passo per SST-2 o AG News; il lavoro è completato quando il tutorial dimostra metriche riproducibili, calibrazione, analisi degli errori e delle slice, reporting e i test indicati.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

feature: missing role: missing size: missing

Title & Overview

Template: Evaluation & Error Analysis: An Intermediate, End-to-End Analysis Tutorial
Overview (≤2 sentences): Learners will design evaluation pipelines that go beyond a single accuracy score, incorporating calibration, confusion analysis, and robustness checks. It is intermediate because it teaches structured error analysis, slice-based evaluation, and reproducibility of metrics.

Purpose

The value-add is building defensible evaluation frameworks that highlight where and why models fail. Learners will explore calibration, error taxonomies, slice-based robustness, and logging reproducible evaluation artifacts for applied NLP projects.

Prerequisites

  • Skills: Python, Git, pandas, ML basics (precision/recall/F1).
  • NLP: classification, token-level tasks, evaluation metrics (macro/micro F1, AUC).
  • Tooling: pandas, scikit-learn, Hugging Face Transformers, torchmetrics/seqeval, MLflow, FastAPI.

Setup Instructions

  • Environment: Conda/Poetry (Python 3.11), deterministic seeds.

  • Install: pandas, scikit-learn, torchmetrics, seqeval, Hugging Face Transformers + Datasets, MLflow, FastAPI.

  • Datasets:

    • Small: SST-2 (sentiment classification).
    • Medium: AG News (topic classification).

  • Repo layout:

    tutorials/t12-eval-error-analysis/
  ├─ notebooks/
  ├─ src/
  │   ├─ metrics.py
  │   ├─ error_analysis.py
  │   ├─ calibration.py
  │   └─ config.yaml
  ├─ data/README.md
  ├─ reports/
  └─ tests/

Core Concepts

  • Metrics granularity: accuracy vs macro/micro F1 vs per-class metrics.
  • Calibration: Expected Calibration Error (ECE), reliability curves.
  • Error taxonomies: confusion pairs, failure cases, robustness slices.
  • Slice-based evaluation: performance by input length, domain, attribute.
  • Reproducibility: log metrics, seeds, configs, and confusion matrices.

Step-by-Step Walkthrough

  1. Data intake & splits: load SST-2/AG News with reproducible splits.

  2. Baseline model predictions: obtain outputs from Logistic Regression and DistilBERT.

  3. Metric computation: accuracy, precision, recall, macro/micro F1, AUC (if binary).

  4. Calibration analysis: compute ECE, plot reliability curves.

  5. Error analysis:

    • Confusion matrices (AG News: misclassifications between Sports/World).
    • Hardest examples (highest loss).
    • Slice performance (short vs long text, with/without punctuation).

  6. Reporting: metrics tables, confusion heatmaps, calibration plots, error slices → reports/t12-eval-error-analysis.md.

  7. (Optional) Serve: FastAPI endpoint that accepts predictions + labels and returns full evaluation report.

Hands-On Exercises

  • Compare micro vs macro F1 for imbalanced datasets.
  • Ablations: with vs without calibration; with vs without slice metrics.
  • Robustness: test on noisy inputs (typos, domain shift).
  • Stretch: implement error taxonomy tags (e.g., ambiguity, OOD, adversarial).

Common Pitfalls & Troubleshooting

  • Metrics misuse: reporting accuracy only on imbalanced datasets.
  • Over-aggregation: averaging hides poor class-level performance.
  • Calibration ignored: models may be accurate but overconfident.
  • Silent errors: mislabeled data mistaken for model errors.
  • Reproducibility gaps: missing config/logging prevents repeatable analysis.

Best Practices

  • Always log confusion matrices, calibration plots, and per-slice metrics.
  • Use MLflow for versioned metric artifacts.
  • Keep Baseline → Error Analysis narrative.
  • Unit tests: ensure metrics consistent under toy datasets.
  • Guardrails: raise warnings for zero-division in F1.

Reflection & Discussion Prompts

  • Why is macro F1 more informative than accuracy in civic datasets?
  • What insights come from calibration beyond precision/recall?
  • How would you design slice metrics for fairness-sensitive applications?

Next Steps / Advanced Extensions

  • Automate evaluation in CI pipelines.
  • Add robustness checks (counterfactuals, adversarial tests).
  • Incorporate fairness metrics (demographic parity, equal opportunity).
  • Lightweight monitoring: online calibration drift.

Glossary / Key Terms

Accuracy, precision, recall, F1, calibration, ECE, reliability curve, confusion matrix, slice metrics.

Additional Resources

Contributors

Author(s): TBD
Reviewer(s): TBD
Maintainer(s): TBD
Date updated: 2025-09-20
Dataset licenses: SST-2 (GLUE), AG News (CC).

Issues Referenced

Epic: HfLA Text Analysis Tutorials (T0–T14).
This sub-issue: T12: Evaluation & Error Analysis.

Lingua principale
Jupyter Notebook
Stelle
33
Fork
23
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di hackforla/data-science

Tutte le issue di hackforla/data-science

Issue simili

Altre issue su Data Engineering

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.