HTML knowledge-base files lose every block boundary: get_text() welds the words together

Start in api/core/rag/extractor/html_extractor.py at HtmlExtractor._load_as_text, then inspect the existing HTML extractor test that currently checks only that characters survive. Run that test and reproduce the provided HTML example; done means block-level boundaries remain separate while inline text keeps its spacing and punctuation.

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report.
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
• Please do not modify this template :) and fill in all the required fields.

Dify version

main (api/core/rag/extractor/html_extractor.py)

Cloud or Self Hosted

Self Hosted (Source)

Steps to reproduce

Upload an ordinary .html file to a knowledge base — or call the extractor
directly:

from core.rag.extractor.html_extractor import HtmlExtractor

# <h1>Quarterly Report</h1><p>Revenue rose.</p><p>Costs fell.</p>
# <ul><li>Item one</li><li>Item two</li></ul>
print(HtmlExtractor("report.html")._load_as_text())

✔️ Expected Behavior

One block per line, the way the page reads:

Quarterly Report
Revenue rose.
Costs fell.
Item one
Item two

❌ Actual Behavior

Quarterly ReportRevenue rose.Costs fell.Item oneItem two

_load_as_text returns soup.get_text(), which concatenates every text node
with nothing in between. A page's block structure lives in the markup, not in the
text, so every boundary disappears and the words on either side are welded into
one: ReportRevenue, fell.Item, oneItem.

That string is what gets chunked and embedded. ReportRevenue is not a word in
any vocabulary, so the chunk that holds the heading is not retrievable by the
heading, and a chunk boundary can now fall in the middle of what used to be two
separate blocks.

The existing test pins only that the characters survive —
"".join(page_content.split()) == "TitleHello" — which is true either way, so
nothing catches it today.

Note for whoever picks this up: the obvious fix is wrong. get_text(" ") or
get_text("\n") also inserts the separator between inline elements, so
<p>Hello <b>world</b>! See <a href="#">this</a>.</p> turns into
Hello world ! See this . The end of each block-level element has to be marked
instead.

I have a patch ready with tests for both halves and will open the PR against this
issue.