A paragraph reading just “Comment” truncates the rest of the article
Maintainers usually reply within 7 days
Assessment
- Difficulty
- 3/5
- Estimated time
- Half a day
- Newbie friendliness
- 55/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- html, typescript
- Domain
- content
Research direction
Start in src/removals/content-patterns.ts: the /^Comments?$/i entry in BOILERPLATE_PATTERNS (around line 19) and the trailing-removal logic near line 1001 that drops everything after a match. Reproduce with the issue's comment.html fixture and debug: true, then check the removal log. Done means a short bold "Comment" paragraph followed by prose keeps the following paragraphs, while real comment sections are still removed. Existing tests for trailing comment removal should keep passing.
Written by the indexing model from the issue text.
Description
Hi kepano, a second one from me, this time relating to reading legal blogs with Defuddle.
Summary
BOILERPLATE_PATTERNS includes /^Comments?$/i. When a short element in the article matches it, everything after it is removed as "trailing non-content". Legal and academic blogs often use a bold "Comment" paragraph to introduce the author's own analysis, so the conclusion of the post is silently dropped.
Context
defuddle0.19.4 (CLI) and 0.19.5 (npx [email protected]), linkedom- macOS 27.0, Node 26.12.0
Reproduction
npx [email protected] parse https://ukhumanrightsblog.com/2024/01/25/financial-sanction-and-free-speech-in-the-high-court/ --md | grep -c "The Court appeared to have little difficulty"
# 0
The page's HTML:
<p class="wp-block-paragraph"><strong>Comment</strong></p>
<p class="wp-block-paragraph">The Court appeared to have little difficulty deciding the Claimant’s application, …</p>
With debug: true, the removals include:
{"step":"removeByContentPattern","reason":"trailing non-content","text":"The Court appeared to have little difficulty deciding the Claimant’s application, …"}
{"step":"removeByContentPattern","reason":"trailing non-content","text":"Returning to the conflict in issue, as the Court recorded, …"}
{"step":"removeByContentPattern","reason":"boilerplate text","text":"Comment"}
Minimal fixture. The page needs some length before the "Comment" paragraph. On a very short page the text survives, I assume because of the retry fallback.
P='<p>The court considered the statutory scheme, the relevant authorities and the evidence filed by both parties. It examined whether the measure was prescribed by law, whether it pursued a legitimate aim, and whether a fair balance had been struck between the rights of the individual and the interests of the community as a whole.</p>'
{ echo '<!doctype html><html><head><title>Example case note</title></head><body><main><article><h1>Example case note</h1>'
for i in 1 2 3 4 5; do echo "$P"; done
echo '<p><strong>Comment</strong></p>'
echo '<p>The court appeared to have little difficulty with the application, but the reasoning raises wider questions that later cases will need to address.</p>'
echo '<p>There are benefits to this approach, but it is not without risks.</p>'
echo '</article></main></body></html>'; } > comment.html
npx [email protected] parse comment.html --md | grep -c -E "little difficulty|There are benefits"
# 0
sed 's#<strong>Comment</strong>#<strong>Analysis</strong>#' comment.html | npx [email protected] parse - --md | grep -c -E "little difficulty|There are benefits"
# 2
Cause
- Pattern: https://github.com/kepano/defuddle/blob/556d74e8c8fd68c49912d4b11a4ae9ec04d7b2b5/src/removals/content-patterns.ts#L19
- Truncation: https://github.com/kepano/defuddle/blob/556d74e8c8fd68c49912d4b11a4ae9ec04d7b2b5/src/removals/content-patterns.ts#L1001
Expected behaviour
A "Comment" heading followed by ordinary prose is kept, along with the paragraphs after it.
Possible approach
Perhaps treat "Comment(s)" as the start of a comments section only when what follows looks like comments? For example comment markup or #comments, rather than ordinary paragraphs.
- Dominant language
- TypeScript
- Stars
- 9.6k
- Forks
- 426
- Avg merge
- 4d 19h
- Merged PRs (30d)
- 6
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from kepano/defuddle
-
extractorType is a minified class name in the browser bundlesPossibly taken @paoValle claimed this 5 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
Maintainers usually reply within 7 days
-
BilibiliExtractor sets the forbidden User-Agent header on its fetches, which makes the transcript download fail in Firefox (CORS preflight -> 405 from the subtitle CDN)Possibly taken @Gopal-Khadka claimed this 10 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 7 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 7 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
kepano/defuddle#310 · 1 comment · 1 reaction ·
Maintainers usually reply within 7 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
Maintainers usually reply within 7 days
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
paperclipai/paperclip#17151 ·
Maintainers usually reply within 1 day
-
needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
AgoraIO-Extensions/agent-infra#3159 ·
Maintainers usually reply within 1 day
-
blocklist addition
Difficulty 1/5 Under an hour Newbie friendliness 66/100
MetaMask/eth-phishing-detect#304596 ·
Maintainers usually reply within 1 day
-
friction
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
kentcdodds/kody#4142 ·
-
[Bug] Statistics | Task statistics are always empty on MikroORM with MySQL or SQLitePossibly taken @NdekoCode claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
ever-co/ever-gauzy#11214 ·
Maintainers usually reply within 1 day