Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[feat] Add data cleaning in `fast-llm prepare`

Abierto
  • ¿Algo no está bien?
#112 3 comentarios 0 reacciones 1 asignado Ver en GitHub

@bigximik ya está trabajando en esto.

Desde el 23/3/2025.

Evaluación

Este issue todavía no se ha evaluado.

Descripción

enhancement need update

🎯 Goal (What & Why)

Add a comprehensive data cleaning stage to the fast-llm prepare command.

fast-llm prepare currently downloads and tokenizes HuggingFace datasets into Fast-LLM's .bin / .idx format using a distributed torchrun setup. However, it performs no data cleaning, which limits training quality and poses risks around PII and malicious content.

This ticket adds a required data cleaning phase that is configurable, fast, and integrated into the distributed preprocessing loop. The goal is to improve model quality, reduce noise, follow best practices (see OLMo-2), and meet responsible AI standards by removing PII and malware from the training corpus.

🚀 Execution Plan

Step 1: What is the smallest working version?
  • Extend fast-llm prepare to apply a modular and configurable cleaning pipeline during preprocessing.
  • All cleaning steps must be integrated into the existing torchrun CPU-only distributed setup, preserving parallelism.
Step 2: Required cleaning filters (all must be implemented):
  • Length filtering
    • Remove documents exceeding a configurable max length (in characters or tokens).
  • n-gram repetition
    • Remove documents with ≥N repeated n-grams (default: 32), as in OLMo-2.
  • Frequency-based filtering
    • Remove documents where:
      • The most frequent word exceeds X% of total tokens (default: 30%).
      • The top-2 most frequent words together exceed Y% of total tokens (default: 50%).
  • Binary content filtering
    • Remove documents that contain mostly binary data.
  • Numerical content filtering
    • Remove documents with a high percentage of numeric tokens (default: configurable threshold, e.g., 50%).
  • PII redaction
    • Integrate Microsoft Presidio for detection and redaction or removal of documents containing sensitive personal information.
  • Malware removal
    • Integrate ClamAV to scan documents and remove any that trigger detections.

All thresholds and filter behaviors must be exposed via the CLI and config files. Document-level logs or counters should be maintained for each filter to aid debugging and analysis.

📌 Acceptance Criteria

  • All listed filters are implemented and integrated into fast-llm prepare.
  • Cleaning is fully configurable, both via CLI and YAML config files.
  • The implementation works with the existing distributed CPU setup (torchrun + Gloo).
  • Performance remains acceptable.
  • Logs report how many documents are removed by each filter.
  • Code is tested and documented.
  • PR includes a performance/impact summary and example CLI usage.

🛠️ Project Management

  • Assign the project to the Fast-LLM project.
  • Set the Estimate field (in days) in the GitHub project.
  • Use the Size field to categorize the PR size (Small/Medium/Large).
  • Assign an owner when opening the issue.
Lenguaje dominante
Python
Estrellas
337
Forks
52
Sin PR fusionados en 30 d

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de ServiceNow/Fast-LLM

Todos los issues de ServiceNow/Fast-LLM

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.