Gateway nodes (Traefik-based reverse proxy) show significantly limited throughput vs. client-side capacity

Open
#2,704 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Active
Tech stack
nginx

Research direction

The payload names two gateway domains, node IDs, and a minimal nginx/WebDAV test rather than repository files. Re-run the single-stream curl download and upload tests through both gateways, then inspect the Traefik path and node network capacity; done means identifying whether the proxy or node capacity causes the throughput ceiling and recording reproducible evidence.

Written by the indexing model from the issue text.

Description

Username on crisp: Peterhulleman
Email id: peterhulleman@pm.me

Describe the bug

Summary:
Investigating a customer report of slow Nextcloud upload/download speeds behind two ThreeFold gateway nodes led to a controlled test that shows both gateways delivering throughput far below the tested capacity of the client-side connection. This suggests a bottleneck in the gateway path itself (Traefik proxy layer or the node's own network capacity), rather than in the customer's local setup or their backend VM.

Background:
Customer (Peter) reported inconsistent Nextcloud upload speeds (160 KB/s–1000 KB/s) via a Nextcloud instance behind two ThreeFold gateway nodes, despite having 25 Mbps of local upload capacity (confirmed via OneDrive uploads reaching 22.4 Mbps). We reproduced the issue independently using a controlled test.

Affected nodes:
Node IDPublic IPv4Assigned domain
6968 178.251.27.20 tfcloud.eu
50 185.69.167.142 gent04.grid.tf

Test methodology:
Deployed a minimal nginx web server on a ThreeFold backend VM, serving a 100 MB test file for download and a WebDAV PUT endpoint for upload.
Registered one domain entry per gateway via the TFGrid dashboard's Domains feature, pointing at the backend VM's Mycelium IP on port 80 (TLS Passthrough disabled, so Traefik terminates TLS and forwards plain HTTP).
Ran single-stream curl download and upload tests from an external client machine, through each gateway's public domain.
Verified the client machine's own connection was not the bottleneck: baseline test (fast.com) showed 61 Mbps download on the client, and the customer had already independently verified 25 Mbps upload on their end via OneDrive.

Results:
DirectionGateway 6968 (tfcloud.eu)Gateway 50 (gent04.grid.tf)
Download 665,071 bytes/sec (0.63 MB/s / 5 Mbps) 510,671 bytes/sec (0.49 MB/s / 4 Mbps)
Upload 0.9–1.05 MB/s (7–8.5 Mbps), varied across repeated runs 0.8–0.96 MB/s (6.5–7.5 Mbps), varied across repeated runs

For reference, client-side capacity confirmed on both ends of the test:
Test client (external machine): 61 Mbps download (fast.com)
Customer's own connection: 25 Mbps upload (OneDrive comparison)
Both are far above the throughput actually delivered through either gateway.

Observations:
Gateway 6968 consistently outperformed gateway 50 in this test 30% faster on download, 10% faster on upload.
Upload throughput varied by 10–15% across repeated runs against the same gateway, suggesting shared/contended capacity rather than a fixed hard cap.
The customer independently reported that speeds improve during evening/night hours, consistent with time-of-day contention on the gateway.
Since neither the test client's connection nor the customer's own connection can explain the throughput ceiling, the limitation appears to sit in the gateway path itself either the Traefik proxy layer or the underlying node's network capacity/allocation.

Test screenshot:

Image Image
Dominant language
Go
Stars
95
Forks
22
Avg merge
2m
Merged PRs (30d)
4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from threefoldtech/zos

All issues in threefoldtech/zos

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.