Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Bug]: preserve_tags/preserve_classes are a no-op for excluded tags (aside, nav, footer, header, form)

Aperta
  • Qualcosa non va?
#2,125 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@Battleplus ci sta già lavorando.

Dal 22/8/2026.

  • #2167 di @Battleplus — chiusa senza merge
  • #2181 di @Battleplus — aperta
  • #2193 di @Battleplus — aperta

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
78/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Tranquilla
Stack tecnologico
python
Ambito
backend

Direzione di ricerca

Start in content_filter_strategy.py, especially filter_content(), _remove_unwanted_tags(), and _is_preserved(). Run the provided HTML snippet without a browser or network, then trace how excluded tags are removed before preservation is checked. Done means preserve_tags and preserve_classes retain matching aside content while the default still excludes it.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

🐞 Bug 🩺 Needs Triage
crawl4ai version

0.9.2

Expected Behavior

PruningContentFilter(preserve_tags=["aside"]) should keep <aside> content in fit_markdown.
#1900 added these options as an escape hatch for content the fit pipeline treats as boilerplate but the user knows is real content.

Same for preserve_classes=["related"] on an <aside class="related">.

Current Behavior

Both options are silently ignored for every tag in excluded_tags (content_filter_strategy.py:101-110): nav, footer, header, aside, script, style, form, iframe, noscript.

filter_content() removes those tags before pruning ever runs:

self._remove_unwanted_tags(soup)   # content_filter_strategy.py:664, decomposes aside
body = soup.find("body")
self._prune_tree(body)             # :668, the only place _is_preserved() is consulted

_remove_unwanted_tags decomposes by tag name and never consults the whitelist:

def _remove_unwanted_tags(self, soup):   # :685
    """Removes unwanted tags"""
    for tag in self.excluded_tags:
        for element in soup.find_all(tag):
            element.decompose()

By the time _is_preserved() (:691) is reached, the node is gone from the tree.
The filter raises nothing and logs nothing.

excluded_tags is hardcoded in RelevantContentFilter.__init__, not a constructor argument, so the only workaround is mutating the instance attribute (f.excluded_tags.discard("aside")).

Is this reproducible?

Yes

Inputs Causing the Bug
- URL(s): none needed, filter_content() takes an HTML string directly
- Settings used: PruningContentFilter(preserve_tags=["aside"])
- Input data: any HTML with an <aside> holding real content
Steps to Reproduce
1. Run the snippet below (no browser, no network).
2. All three cases print "aside kept: False".
3. Expected: True for the preserve_tags and preserve_classes cases.
Code snippets
from crawl4ai.content_filter_strategy import PruningContentFilter

html = (
    "<html><body>"
    "<article><p>" + "Main article body with plenty of words to pass the pruning threshold easily. " * 8 + "</p></article>"
    '<aside class="related"><h2>Related reading</h2><p>'
    + "This sidebar note is real content the author wrote and wants kept. " * 8
    + "</p></aside>"
    "</body></html>"
)

for kwargs in ({}, {"preserve_tags": ["aside"]}, {"preserve_classes": ["related"]}):
    out = " ".join(PruningContentFilter(**kwargs).filter_content(html))
    print(kwargs, "-> aside kept:", "Related reading" in out)

# 0.9.2 / develop @ 2d8f673:
# {}                                -> aside kept: False   # correct, that's the default
# {'preserve_tags': ['aside']}      -> aside kept: False   # BUG
# {'preserve_classes': ['related']} -> aside kept: False   # BUG
OS

macOS

Python version

3.12

Browser

N/A

Browser version

No response

Error logs & Screenshots (if applicable)

No response

Lingua principale
Python
Stelle
85.4k
Fork
8.9k
Merge medio
14h 10m
PR unite (30g)
12

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di unclecode/crawl4ai

Tutte le issue di unclecode/crawl4ai

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.