[Bug]: Crawler storage access silently breaks when cloud storage endpoint resolves to a private IP -- blocked by default `crawler-limit-egress` NetworkPolicy
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 22/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- helm, kubernetes
- Domain
- infrastructure, networking
Research direction
Start with the default crawler-limit-egress policy in chart/templates/networkpolicies.yaml and the label that selects crawler pods in chart/app-templates/crawler.yaml. The open question is whether the policy's 10.0.0.0/8 exception is too broad for Spaces endpoints that resolve to private IPs in some regions, and the issue leaves the DNS and Cilium cause unconfirmed. Done means a documented or tested fix that lets crawler storage access work when the endpoint resolves to a private address, which needs maintainer input on the intended policy.
Written by the indexing model from the issue text.
Description
Browsertrix Version
v1.26.1-21bd6c6
Browsertrix Host
Self-Hosted
What did you expect to happen? What happened instead?
Expected: Crawls continue to work after a Kubernetes upgrade.
What happened: After upgrading DOKS in the syd1 region, crawler pods cannot reach the object storage endpoint. Any crawler operation that touches storage is affected -- writing WARCs, uploading the finished WACZ, reading storage-backed seeds, and downloading browser profiles. The browser profile download is the first storage operation at startup, so that is where the failure surfaces. The crawl pod appears to start normally before dying with a fatal error during profile download:
{"logLevel":"fatal","context":"browser","message":"Profile setup failed. Quitting",
"details":{"message":"connect ETIMEDOUT 10.126.15.254:443"}}
There is no warning during helm upgrade and no obvious connection between the failure and the NetworkPolicy in the logs.
Reproduction instructions
- Deploy Browsertrix on DOKS (syd1 region, but potentially all) with external object storage (Spaces)
- Upgrade the DOKS cluster from Kubernetes 1.34.x to 1.35.x (this upgrades Cilium from v1.18 to v1.19 -- see DigitalOcean Kubernetes changelog)
- Start any crawl
- If using a browser profile, observe
ETIMEDOUTin crawler pod logs after several minutes; otherwise the failure will surface at the first WARC upload
Screenshots / Video
No response
Environment
No response
Additional details
The storage endpoint syd1.digitaloceanspaces.com resolves to a private IP from within crawler pods in the syd1 cluster:
Name: syd1.digitaloceanspaces.com
Address: 10.126.15.254
10.126.15.254 falls within 10.0.0.0/8, which is excluded from egress in the default crawler-limit-egress NetworkPolicy defined in chart/templates/networkpolicies.yaml:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
This policy applies to all pods labelled network-policy: limit-crawler-egress, as set in chart/app-templates/crawler.yaml.
For comparison, a DOKS cluster in the sfo3 region (Cilium v1.18.3, pre-upgrade) resolves sfo3.digitaloceanspaces.com to a public IP and crawls work correctly:
Name: sfo3.digitaloceanspaces.com
Address: 138.68.34.161
The root cause of the DNS difference is unconfirmed -- it may be a regional difference in how DigitalOcean serves Spaces endpoints (private in syd1, public in sfo3), a Cilium version difference (1.18.3 vs 1.19.8), or both. What is confirmed: when the Spaces hostname resolves to a private IP within 10.0.0.0/8, the default NetworkPolicy silently blocks all crawler storage access.
- Dominant language
- TypeScript
- Stars
- 502
- Forks
- 83
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 32
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from webrecorder/browsertrix
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
webrecorder/browsertrix#3823 ·
Maintainers usually reply within 1 day
-
[Bug]: DigitalOcean ansible playbook deploys masked MongoDB password (********) with doctl >= 2.0.0Possibly taken @axllent claimed this today. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
webrecorder/browsertrix#3815 · 2 comments ·
Maintainers usually reply within 1 day
-
enhancement idea
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
webrecorder/browsertrix#3715 ·
Maintainers usually reply within 1 day
-
bug front end
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
webrecorder/browsertrix#3708 ·
Maintainers usually reply within 1 day
-
back end
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
webrecorder/browsertrix#3626 ·
Maintainers usually reply within 1 day
All issues in webrecorder/browsertrix
Similar issues
-
bug documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Potential Bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Comfy-Org/ComfyUI_frontend#23773 ·
Maintainers usually reply within 1 day
-
community first-timers-only good first issue hacktoberfest help wanted low hanging fruit up-for-grabs
Difficulty 1/5 Under an hour Newbie friendliness 85/100
lingdojo/kana-dojo#34301 · 1 comment · 5 reactions ·
Maintainers usually reply within 1 day
-
code-quality refactoring
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
github/gh-aw-firewall#11797 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
code-yeongyu/senpi#7007 ·
Maintainers usually reply within 1 day