[Bug]: Crawler storage access silently breaks when cloud storage endpoint resolves to a private IP -- blocked by default `crawler-limit-egress` NetworkPolicy

Open
#3,732 3 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
22/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
helm, kubernetes

Research direction

Start with the default crawler-limit-egress policy in chart/templates/networkpolicies.yaml and the label that selects crawler pods in chart/app-templates/crawler.yaml. The open question is whether the policy's 10.0.0.0/8 exception is too broad for Spaces endpoints that resolve to private IPs in some regions, and the issue leaves the DNS and Cilium cause unconfirmed. Done means a documented or tested fix that lets crawler storage access work when the endpoint resolves to a private address, which needs maintainer input on the intended policy.

Written by the indexing model from the issue text.

Description

bug
Browsertrix Version

v1.26.1-21bd6c6

Browsertrix Host

Self-Hosted

What did you expect to happen? What happened instead?

Expected: Crawls continue to work after a Kubernetes upgrade.

What happened: After upgrading DOKS in the syd1 region, crawler pods cannot reach the object storage endpoint. Any crawler operation that touches storage is affected -- writing WARCs, uploading the finished WACZ, reading storage-backed seeds, and downloading browser profiles. The browser profile download is the first storage operation at startup, so that is where the failure surfaces. The crawl pod appears to start normally before dying with a fatal error during profile download:

{"logLevel":"fatal","context":"browser","message":"Profile setup failed. Quitting",
 "details":{"message":"connect ETIMEDOUT 10.126.15.254:443"}}

There is no warning during helm upgrade and no obvious connection between the failure and the NetworkPolicy in the logs.

Reproduction instructions
  1. Deploy Browsertrix on DOKS (syd1 region, but potentially all) with external object storage (Spaces)
  2. Upgrade the DOKS cluster from Kubernetes 1.34.x to 1.35.x (this upgrades Cilium from v1.18 to v1.19 -- see DigitalOcean Kubernetes changelog)
  3. Start any crawl
  4. If using a browser profile, observe ETIMEDOUT in crawler pod logs after several minutes; otherwise the failure will surface at the first WARC upload
Screenshots / Video

No response

Environment

No response

Additional details

The storage endpoint syd1.digitaloceanspaces.com resolves to a private IP from within crawler pods in the syd1 cluster:

Name:   syd1.digitaloceanspaces.com
Address: 10.126.15.254

10.126.15.254 falls within 10.0.0.0/8, which is excluded from egress in the default crawler-limit-egress NetworkPolicy defined in chart/templates/networkpolicies.yaml:

- to:
  - ipBlock:
      cidr: 0.0.0.0/0
      except:
      - 10.0.0.0/8
      - 172.16.0.0/12
      - 192.168.0.0/16

This policy applies to all pods labelled network-policy: limit-crawler-egress, as set in chart/app-templates/crawler.yaml.

For comparison, a DOKS cluster in the sfo3 region (Cilium v1.18.3, pre-upgrade) resolves sfo3.digitaloceanspaces.com to a public IP and crawls work correctly:

Name:   sfo3.digitaloceanspaces.com
Address: 138.68.34.161

The root cause of the DNS difference is unconfirmed -- it may be a regional difference in how DigitalOcean serves Spaces endpoints (private in syd1, public in sfo3), a Cilium version difference (1.18.3 vs 1.19.8), or both. What is confirmed: when the Spaces hostname resolves to a private IP within 10.0.0.0/8, the default NetworkPolicy silently blocks all crawler storage access.

Dominant language
TypeScript
Stars
502
Forks
83
Avg merge
2d 14h
Merged PRs (30d)
32

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from webrecorder/browsertrix

All issues in webrecorder/browsertrix

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.