Feature Request: Auto-Follow Single-Frame Framesets

Start with async_configs.py to trace CrawlerRunConfig, then follow the crawler strategy's _crawl_web flow around Phase 4 overlay removal and Phase 5 HTML capture. Verify how page.frames, frame content, and redirected_url are represented before implementing the requested single-frame behavior. Done means qualifying framesets return the child frame content and URL while non-qualifying pages and follow_frames=False retain normal capture.

Summary

When crawling a URL that serves an HTML <frameset> page, crawl4ai returns the empty frameset shell instead of the actual content, then incorrectly reports the result as blocked by antibot protection. The URL is perfectly accessible — the content simply lives inside the frame, which Playwright has already loaded but crawl4ai does not read.

---

Reproduction

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, BrowserConfig
import asyncio

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://www.variosolar.de/")
        print(result.success)        # False
        print(result.error_message)  # "Blocked by anti-bot protection: ..."
        print(len(result.html))      # 612

asyncio.run(main())

Output:

[ERROR] Blocked by anti-bot protection: Structural: minimal_text on small page (612 bytes, 32 chars visible)
False
612

Opening https://www.variosolar.de/ in a browser shows a fully rendered, content-rich page with no antibot challenge whatsoever.

---

Root Cause

variosolar.de responds with this HTML:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Frameset//EN" ...>
<html>
  <head><title>VarioSolar</title></head>
  <frameset rows="100%">
    <frame src="https://www.varioself.de/bauweise/varioenergy/" frameborder="0" noresize="noresize">
  </frameset>
</html>

This is a frameset redirect — a pattern from the early web where a site acts as a transparent wrapper pointing all traffic to another URL. A browser renders it invisibly: the user sees the frame content, the frameset shell is never visible.

crawl4ai calls page.content() on the top-level page after load, which returns the frameset HTML (612 bytes). The frame content — which Playwright has already fully loaded in a child frame — is never read. The antibot detector then sees a nearly empty page and fires a false positive.

---

Why Existing Config Options Don't Help

Option	Why it fails
process_iframes=True	Handles <iframe> elements only. <frameset>/<frame> are a distinct, older HTML construct not touched by this code path.
remove_overlay_elements=True	Irrelevant — the problem is missing content, not overlays.
wait_until="networkidle"	The frameset shell loads instantly. Waiting longer returns the same 612 bytes.
delay_before_return_html	Same reason — time does not change what page.content() returns.

Workaround (manual): Inspect the raw HTML of the frameset page, extract the src attribute from the <frame> tag, and crawl that URL directly:

# Instead of the user-facing URL:
result = await crawler.arun("https://www.variosolar.de/")          # ✗ empty shell

# Crawl the actual framed URL directly:
result = await crawler.arun("https://www.varioself.de/bauweise/varioenergy/")  # ✓ works

This is a poor user experience: the site looks completely normal in a browser, the error message is actively misleading, and finding the real URL requires manually inspecting raw HTML.

---

Proposed Fix: follow_frames on CrawlerRunConfig

Behaviour

Add a follow_frames parameter to CrawlerRunConfig. When enabled, crawl4ai detects single full-viewport frameset pages after load and transparently returns the child frame's content and URL — exactly as HTTP 301/302 redirects are already followed automatically.

The key insight: Playwright has already loaded and rendered the frame content. It is accessible via page.frames at no extra cost. No second HTTP request is needed.

Detection Heuristic

Qualify a page as a frameset redirect only if all conditions hold:

1. page.frames has exactly 2 entries (main frame + one child)
2. page.content() contains a <frameset> tag
3. The frameset has rows="100%", cols="100%", rows="*", cols="*", or no explicit rows/cols attribute (all equivalent to a single full-viewport frame)

This is conservative: multi-frame layouts (nav + content panes) have more than 2 frames and are never auto-followed.

Implementation Sketch

1. Add follow_frames to CrawlerRunConfig:

# async_configs.py
follow_frames: bool = True  # in __init__ signature
self.follow_frames = follow_frames  # in __init__ body
"follow_frames": self.follow_frames  # in to_dict()

2. Add a detection helper method to the crawler strategy:

async def _try_follow_frameset(self, page: Page) -> tuple:
    """
    If the page is a single full-viewport frameset, return the child frame's
    (html, url). Otherwise return (None, None).
    No extra HTTP request — Playwright has already loaded the frame.
    """
    import re
    try:
        frames = page.frames
        if len(frames) != 2:
            return None, None

        main_html = await page.content()
        if "<frameset" not in main_html.lower():
            return None, None

        match = re.search(r'<frameset([^>]*)>', main_html, re.IGNORECASE)
        if not match:
            return None, None

        attrs = match.group(1).lower()
        rows = re.search(r'rows=["\']([^"\']+)["\']', attrs)
        cols = re.search(r'cols=["\']([^"\']+)["\']', attrs)

        def is_full_viewport(val: str) -> bool:
            parts = [p.strip() for p in val.split(",")]
            return len(parts) == 1 and parts[0] in ("100%", "*")

        if rows and not is_full_viewport(rows.group(1)):
            return None, None
        if cols and not is_full_viewport(cols.group(1)):
            return None, None

        child_frame = frames[1]
        child_html = await child_frame.content()
        child_url = child_frame.url

        if not child_html or not child_url:
            return None, None

        return child_html, child_url

    except Exception:
        return None, None

3. Call it in _crawl_web after Phase 4 (overlay removal), before Phase 5 (HTML capture):

# Phase 4.5: Frameset follow
if config.follow_frames:
    frame_html, frame_url = await self._try_follow_frameset(page)
    if frame_html is not None:
        # Surface the frame URL the same way HTTP redirects are surfaced
        return AsyncCrawlResponse(
            html=frame_html,
            response_headers=response_headers,
            status_code=status_code,
            redirected_url=frame_url,
            redirected_status_code=redirected_status_code,
        )

# Phase 5: normal HTML capture
html = await page.content()

Expected behaviour after fix

result = await crawler.arun("https://www.variosolar.de/")
print(result.success)        # True
print(result.redirected_url) # "https://www.varioself.de/bauweise/varioenergy/"
print(len(result.html))      # ~28000 (full page content)

---

Why Not Default-On?

A full-viewport single-frame frameset has no meaningful use as raw output — the shell contains zero content by design. However, silently crawling a different URL (and potentially a different domain) than the one provided could be unexpected in domain-scoped crawls or URL validation workflows. Recommended path:

• Ship as follow_frames=True (default on) since the frameset shell is never the desired output
• Document follow_frames=False as the escape hatch for users who explicitly want the raw frameset response

Why Not Extend process_iframes?

<iframe> and <frameset>/<frame> are fundamentally different:

	<iframe>	<frameset>/<frame>
Part of the parent DOM	Yes	No — separate browsing context
process_iframes handles it	Yes	No
Deprecated in HTML5	No	Yes
Typical use	Embedded widgets	Full-page redirect wrapper
Desired crawl behaviour	Inline content into page	Replace page content entirely

These warrant separate flags with separate semantics.