Add include/exclude pattern support to MCP scrape_docs tool

Start in src/mcp/mcpServer.ts, where the scrape_docs tool schema and ScrapeTool.execute() options are defined. Review the existing include/exclude pattern support in ScrapeTool and src/scraper/utils/patternMatcher.ts, then verify that both optional pattern arrays are exposed to MCP clients and passed through correctly.

Summary

The MCP server's scrape_docs tool currently lacks support for include/exclude patterns, while both the CLI and Web UI already have full pattern support. This creates an inconsistency where MCP clients cannot leverage the powerful pattern filtering capabilities that are available in other interfaces.

Current State

✅ Already Implemented

• CLI: Supports multiple --include-pattern and --exclude-pattern flags
• Web UI: Supports multi-line textareas with newline or comma-separated patterns
• Core Engine: Robust pattern matching with glob and regex support, OR logic, and exclude precedence

❌ Missing

• MCP Server: The scrape_docs tool in src/mcp/mcpServer.ts doesn't expose includePatterns or excludePatterns parameters

Proposed Solution

Add the following optional parameters to the scrape_docs tool schema:

includePatterns: z
  .array(z.string())
  .optional()
  .describe("Glob or regex patterns for URLs to include. Regex patterns must be wrapped in slashes, e.g. /pattern/."),
excludePatterns: z
  .array(z.string())
  .optional()
  .describe("Glob or regex patterns for URLs to exclude. Exclude takes precedence over include. Regex patterns must be wrapped in slashes, e.g. /pattern/."),

And pass them through to the ScrapeTool.execute() call in the options.

Technical Details

• The underlying ScrapeTool class already supports these parameters
• Pattern matching logic is implemented in src/scraper/utils/patternMatcher.ts
• Supports both glob patterns (via minimatch) and regex patterns (wrapped in /pattern/)
• Multiple patterns work with OR logic (any pattern match wins)
• Exclude patterns take precedence over include patterns

Files to Modify

• src/mcp/mcpServer.ts - Add parameters to the tool schema and pass them to ScrapeTool

Benefits

• Provides feature parity across all interfaces (CLI, Web UI, MCP)
• Enables MCP clients (Claude, Cline, etc.) to perform targeted documentation scraping
• Leverages existing, well-tested pattern matching infrastructure

Related

This maintains consistency with the existing pattern support documented in:

• CLI help text for --include-pattern and --exclude-pattern flags
• Web UI tooltips explaining pattern syntax and multi-line support