[BUG]: Article cleaner removes boilerplate phrases from inside article sentences
Evaluación
- Dificultad
- 2/5
- Tiempo estimado
- 1-3 horas
- Aptitud para principiantes
- 25/100
Línea de trabajo
Start in backend/app/modules/scraper/cleaner.py at clean_extracted_text; run the reproduction snippet to see the sentence truncation and the about us word cut. Compare how phrases ending in .* and short phrases like about us are matched, then restrict removal to whole boilerplate lines. Add tests covering lines that should be removed and article sentences that must survive; done when the snippet returns both sentences intact.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Description
clean_extracted_text in backend/app/modules/scraper/cleaner.py removes its boilerplate phrases wherever they appear in the article, not only on boilerplate lines. So it also changes the article text itself:
- Phrases that end in
.*remove the rest of the sentence. An article line like "An engine powered by hydrogen could drive a car for five hundred kilometres." becomes "An engine", because ofpowered by .*. The same happens withphoto by.*,read more at.*,disclaimer:.*and the others. - Short phrases are cut out of words.
about usturns "Readers asked about using the new tool" into "Readers asked ing the new tool".
The cleaned text is what the bias check, fact check and perspective steps get, so they work on a damaged article.
Steps to reproduce
from app.modules.scraper.cleaner import clean_extracted_text
print(clean_extracted_text(
"An engine powered by hydrogen could drive a car for five hundred kilometres.\n"
"Readers asked about using the new tool in newsrooms around the world."
))
The first sentence is reduced to "An engine" (and then dropped as too short), and the second loses "about us".
Expected
Only lines that are boilerplate (for example a line starting with "Read more at" or "Sign up for our newsletter") are removed, and article sentences stay as they are.
I have a fix with tests ready and will open a PR.
- Lenguaje dominante
- TypeScript
- Estrellas
- 58
- Forks
- 90
- Merge medio
- 11 h 51 min
- PR fusionados (30 d)
- 3
Preparar el entorno
- Sin Dockerfile ni archivo de Docker Compose
- Sin plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de AOSSIE-Org/Perspective
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 83/100
AOSSIE-Org/Perspective#191 ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 85/100
AOSSIE-Org/Perspective#190 ·
-
[BUG]: Missing input validation in URlRequest and ChatQuery causes 500 crashes and wasteful scraper executionPosiblemente ocupada @agarwalpranav0711 la tomó hace 25 días. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
AOSSIE-Org/Perspective#178 ·
-
[BUG]: chunk_rag_data crashes with ValueError when perspective is a dict, silently dropping RAG vector storagePosiblemente ocupada @agarwalpranav0711 la tomó hace 36 días. Abiertobug
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
AOSSIE-Org/Perspective#175 ·
-
[BUG]: Scraper passes headers as query params in requests.get (User-Agent never sent)Posiblemente ocupada @Th-Shivam la tomó hace 50 días. Abiertobug
Dificultad 1/5 Menos de una hora Aptitud para principiantes 92/100
AOSSIE-Org/Perspective#161 · 1 comentario ·
Todos los issues de AOSSIE-Org/Perspective
Issues similares
-
documentation lens:human needs-triage
Dificultad 2/5 1-3 horas Aptitud para principiantes 73/100
thebristolsound/birdbrain#2246 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 1/5 Menos de una hora Aptitud para principiantes 78/100
Los mantenedores suelen responder en 1 día
-
A voting window past about 69.9 million hours stores a deadline that reads back as no deadlineAbiertobug ready-for-human
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
BoardGamesEmpire/board-games-empire-backend#1452 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
good first issue stretch
Dificultad 2/5 1-3 horas Aptitud para principiantes 65/100
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
Los mantenedores suelen responder en 1 día