Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Bug]: MCP scrape tools lack wait_until / SPA support that REST API and CLI provide

Aperta
  • Qualcosa non va?
#1,963 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
52/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Tranquilla
Stack tecnologico
python
Ambito
api, documentation

Direzione di ricerca

Start at the MCP scrape tool schemas and compare their exposed arguments with the existing crawler_config options used by POST /crawl and crwl -c. Verify the MCP tools on a JavaScript-heavy page, then confirm they accept wait_until and delay_before_return_html and that the documentation explains default wait behavior across MCP, REST, and CLI.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

⚙️ In-progress 🐞 Bug 📌 Root caused
crawl4ai version

0.8.6

Expected Behavior

MCP scrape tools (crawl4ai_md, crawl4ai_html, etc.) should accept the same wait_until, delay_before_return_html, and cache_mode parameters as the REST API (POST /crawl) and CLI (crwl -c). This would allow MCP-based agents to reliably scrape JavaScript-heavy pages by waiting for content to fully load.

Example desired usage:

{
  "url": "https://example.com/dynamic-content",
  "wait_until": "networkidle",
  "delay_before_return_html": 2
}

Environment

Component Version
Crawl4AI 0.8.6
OS Debian GNU/Linux 12 (bookworm)
Python 3.12.13
Image unclecode/crawl4ai:latest

Suggested Fix

  1. Expose crawler_config parameters on MCP tool schemas — map wait_until, delay_before_return_html, cache_mode, etc. to the existing REST/CLI options
  2. Document MCP defaults vs REST/CLI behavior

Acceptance Criteria

  • MCP scrape tools accept wait_until parameter (load, domcontentloaded, networkidle, commit)
  • MCP scrape tools accept delay_before_return_html parameter
  • Documentation clarifies default wait behavior for each interface (MCP vs REST vs CLI)
Current Behavior

MCP tools return immediately after initial DOM load, without waiting for dynamic content. No parameters are exposed to control wait behavior.

  • REST API with crawler_config.wait_until: "networkidle" → ✅ Complete rendered content
  • CLI with -c 'wait_until=networkidle' → ✅ Complete rendered content
  • MCP tools → ❌ Incomplete content (dynamic elements missing)

Current workaround: Bypass MCP entirely and use POST /crawl directly.

Is this reproducible?

Yes

Inputs Causing the Bug
Any JavaScript-heavy / AJAX-driven page where content loads after initial page load.
Steps to Reproduce
1. Start Crawl4AI with MCP enabled at `/mcp/sse`
2. Configure any MCP client (e.g., OpenCode) with the Crawl4AI MCP server
3. Call MCP scrape tool:
   
   crawl4ai_md(url="https://example.com/dynamic-content")
   
4. Observe: Content is incomplete (dynamic elements not rendered)
5. Compare with working REST call:
   
   curl -s http://localhost:11235/crawl \
     -H 'Content-Type: application/json' \
     -d '{
       "urls": ["https://example.com/dynamic-content"],
       "crawler_config": {
         "wait_until": "networkidle",
         "cache_mode": "bypass"
       }
     }'
   
6. Observe: REST returns complete rendered content
Code snippets
OS

Debian GNU/Linux 12 (bookworm)

Python version

3.12.13

Browser

No response

Browser version

No response

Error logs & Screenshots (if applicable)

No response

Lingua principale
Python
Stelle
84.5k
Fork
8.7k
Merge medio
3g 9h
PR unite (30g)
17

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di unclecode/crawl4ai

Tutte le issue di unclecode/crawl4ai

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.