[BUG]: Article cleaner removes boilerplate phrases from inside article sentences

Abierto
#181 1 comentario 0 reacciones 0 asignados Ver en GitHub

@itsnjstyle27 ya está trabajando en esto.

Desde el 30/10/2026.

  • #182 de @itsnjstyle27 — abierto

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
25/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
backend

Línea de trabajo

Start in backend/app/modules/scraper/cleaner.py at clean_extracted_text; run the reproduction snippet to see the sentence truncation and the about us word cut. Compare how phrases ending in .* and short phrases like about us are matched, then restrict removal to whole boilerplate lines. Add tests covering lines that should be removed and article sentences that must survive; done when the snippet returns both sentences intact.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Description

clean_extracted_text in backend/app/modules/scraper/cleaner.py removes its boilerplate phrases wherever they appear in the article, not only on boilerplate lines. So it also changes the article text itself:

  • Phrases that end in .* remove the rest of the sentence. An article line like "An engine powered by hydrogen could drive a car for five hundred kilometres." becomes "An engine", because of powered by .*. The same happens with photo by.*, read more at.*, disclaimer:.* and the others.
  • Short phrases are cut out of words. about us turns "Readers asked about using the new tool" into "Readers asked ing the new tool".

The cleaned text is what the bias check, fact check and perspective steps get, so they work on a damaged article.

Steps to reproduce

from app.modules.scraper.cleaner import clean_extracted_text

print(clean_extracted_text(
    "An engine powered by hydrogen could drive a car for five hundred kilometres.\n"
    "Readers asked about using the new tool in newsrooms around the world."
))

The first sentence is reduced to "An engine" (and then dropped as too short), and the second loses "about us".

Expected

Only lines that are boilerplate (for example a line starting with "Read more at" or "Sign up for our newsletter") are removed, and article sentences stay as they are.

I have a fix with tests ready and will open a PR.

Lenguaje dominante
TypeScript
Estrellas
58
Forks
90
Merge medio
11 h 51 min
PR fusionados (30 d)
3

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de AOSSIE-Org/Perspective

Todos los issues de AOSSIE-Org/Perspective

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.