Feature Proposal: RTK Web Hook for HTML-to-Markdown Compression

Hi RTK team,

I’d like to propose a feature that extends RTK’s core idea beyond terminal-heavy developer commands into web content retrieval.

RTK already provides strong value by filtering and compressing command output before it reaches the LLM context window. A natural next step would be to apply the same principle to web pages fetched by agents via commands like curl, fetch, or similar tooling.

Problem

When an AI agent fetches a web page directly, it often receives a very large HTML document full of noise:

• <script> tags
• <style> blocks
• navigation and boilerplate
• long URLs
• repeated layout markup
• irrelevant DOM sections

This consumes a large number of tokens while adding very little useful meaning for the model.

In practice, the agent usually needs only the readable page content, not the full raw HTML.

Proposed Feature

Add an RTK Web Hook that can intercept web-page fetches and transform HTML into a compact, LLM-friendly representation before it reaches the model.

High-level pipeline

1. Detect that the response is HTML or webpage-like content.
2. Strip obvious non-content elements such as:
   • script
   • style
   • iframe
3. Extract the main readable content where possible:
   • main
   • article
   • other high-signal content containers
4. Convert the remaining content into clean Markdown.
5. Apply RTK-style compression:
   • collapse excessive whitespace
   • normalize repeated blank lines
   • shorten repeated long links into references like [1], [2]
   • preserve headings, lists, code blocks, and links

Example

Instead of:

curl https://example.com

returning a large raw HTML payload to the agent, the effective result could become something closer to:

# Article Title

Short readable content here.

## Section
Relevant body text here.

[1] https://example.com/some/very/long/url

So a page that might cost thousands of tokens as raw HTML could become a few hundred tokens of useful content.

Why This Fits RTK

This feature follows the same general value proposition RTK already provides for terminal output:

• reduce token usage
• improve signal-to-noise ratio
• preserve useful structure
• make agent context more efficient

Web pages are another major source of token waste, especially for agent workflows that browse docs, blog posts, issues, reference pages, and articles.

Important Constraint

This should not rewrite every curl response blindly.

The feature should only activate when the fetched content is clearly HTML / webpage content, or when the user explicitly opts in.

It should not modify:

• JSON API responses
• XML
• binary content
• plain text outputs that are already clean

Suggested v1 Scope

A safe first version could be intentionally narrow:

• only process responses that look like text/html
• do not touch JSON, XML, binary, or plain API responses
• preserve raw output on parse failure
• allow explicit opt-in or opt-out via config
• possibly start as an explicit command path instead of transparent rewriting

Possible UX Options

Option A — explicit command

rtk web https://example.com

Pros

• safest rollout
• easy to test and benchmark
• no risk of breaking existing curl workflows

Cons

• less transparent than the normal RTK experience

Option B — selective hook rewrite

Intercept curl or similar commands only when the target appears to be a webpage and the response type is HTML.

Pros

• very convenient
• feels native to RTK’s hook model

Cons

• harder to get correctness right
• needs careful heuristics and fallback behavior

Option C — opt-in config

[web]
enabled = true
mode = "html_only"

Pros

• controlled rollout
• easy for advanced users
• avoids surprising existing users

My preference would be Option A or C first, then consider automatic rewriting later.

Suggested Implementation Sketch

A possible internal pipeline for v1:

1. Execute the fetch command normally.

2. Inspect response headers and/or content sniffing.

3. If non-HTML: return original output untouched.

4. If HTML:

   • parse document
   • remove low-value nodes
   • extract main content candidates
   • convert to Markdown
   • compress whitespace and links

5. If anything fails:

   • return original output unchanged

Token-Saving Opportunities

Possible compression steps:

• DOM pruning before conversion
• main-content extraction before full Markdown generation
• URL deduplication and numeric link references
• collapse repeated whitespace
• remove empty sections and decorative fragments

Non-goals for v1

To keep the first version reviewable, I would explicitly avoid:

• JavaScript rendering / headless browser execution
• full reader-mode heuristics for every site
• handling authentication-heavy sites
• transforming API JSON responses
• automatic summarization by an LLM

Why I Think This Is Worth Building

Web pages are a strong source of context waste in agent workflows.

A lightweight HTML-to-Markdown compression path would let agents read more pages within the same context window, reduce cost, and improve the quality of the text that reaches the model.

This feels like a natural extension of RTK’s existing philosophy: remove junk, keep signal, and minimize token waste.

Willing to Implement

I’d be happy to implement this myself if the direction makes sense to maintainers.

My suggested rollout would be:

1. align on the interface
2. implement a narrow HTML-only v1
3. add tests and benchmarks
4. document behavior and configuration
5. discuss transparent hook-based rewriting later

If this sounds aligned, I can open a draft PR or start with a design sketch for the v1 interface and pipeline.