zeroclaw-labs/zeroclaw
View on GitHub[Feature]: Propose SearXNG search support and improve Web Search robustness
Open
#5316 opened on Apr 5, 2026
configenhancementhelp wantedpriority:p2risk: highstatus:acceptedstatus:no-staletooltool:web
Description
Summary
Add support for SearXNG as a privacy-focused search provider and improve the overall reliability of the web search tool by adding CAPTCHA detection for DuckDuckGo.
Problem statement
The current web search tool is limited in several ways that affect autonomous agents:
- DuckDuckGo CAPTCHAs: DuckDuckGo often blocks requests from automated environments (e.g., Docker, cloud VMs), silently returning empty results instead of reporting the block. The tool currently lacks explicit detection of these blocks, leading to silent failures.
- Limited Provider Options: Users in restricted or privacy-conscious environments need a self-hosted search alternative. Currently, only DuckDuckGo (scraping) and Brave (API) are supported, but there is no option for a self-hosted metasearch engine.
- Resilience: The tool lacks configurable retries and health checks for search providers, which is critical for long-running autonomous tasks.
Proposed solution
A phased rollout of the following improvements:
- Configuration Extensions: Add fields to
WebSearchConfig for SearXNG authentication, language, engines, safesearch, and retry limits. Ensure
SEARXNG_INSTANCE_URLcan be set via environment variable. - Robust CAPTCHA Detection: Update the DuckDuckGo provider to detect 403 Forbidden status and verification redirect URLs (
wr.do?u=), providing actionable feedback to the user. - SearXNG Support Enhancements: Implement exponential backoff for retries, a lazy network health check on first use, and improved result parsing (including source engines and published dates).
- Type-safe Provider Routing: Introduce a SearchProvider enum and refactor routing logic to use this enum, simplifying aliases and fallback management.
Non-goals / out of scope
No response
Alternatives considered
No response
Acceptance criteria
- Config keys for SearXNG are documented and validated.
- DuckDuckGo blocks are explicitly reported with a suggestion to switch providers.
- SearXNG provider supports
SEARXNG_INSTANCE_URLenv var and advanced parameters. - Behavior is verified under both normal and blocked/error conditions.
Architecture impact
- src/config/schema.rs (Config schema updates)
- src/tools/web_search_tool.rs (Search logic and parsing)
- src/tools/web_search_provider_routing.rs (Routing refactor)
- src/tools/mod.rs (Tool initialization)
Risk and rollback
No response
Breaking change?
No
Data hygiene checks
- I removed personal/sensitive data from examples, payloads, and logs.
- I used neutral, project-focused wording and placeholders.