COG window from a slow server fails: a multi-MB tile is dropped by the 30 s tile-fetch timeout

Open
#2,570 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
62/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
rust

Research direction

Start in crates/tiles/src/cog.rs at read, where the request goes through cache::fetch_counted_with_ceiling, then look at agent() and WINDOW_WAIT in cache.rs. Add a second shared agent for COG range reads over 1 MB, with the 5-minute global timeout, and keep the 30-second agent for smaller reads and map tiles. Done when a cog.rs test serving a 2 MB tile over about 35 s reads successfully, and a 20 KB tile from a stalled server still fails at 30 s.

Written by the indexing model from the issue text.

Description

Before you start
  • I searched the existing issues and did not find a duplicate.
  • This is a bug report, not a usage question.
What happened?

A COG window from a slow server fails while its tiles are still arriving. The error is:

https://ceph.opengeohub.org/arco/soildb/soildbpc_ccwpml_m_30m_bg0..30cm_20000101_20051231_go01_epsg4326_v1.2.0_srl04.tif: timeout: global

COG reads go through the tile-fetch workers (crates/tiles/src/cog.rs read → cache::fetch_counted_with_ceiling). Those workers share the agent from cache.rs agent(), which has a 30-second global timeout per request. That suits a 20 KB map tile. A COG tile, though, is often a few MB: OpenGeoHub's global layers use 2048 × 2048 tiles of 4 to 8 MB.

At the 80 to 450 KB/s their servers give from Brazil, one tile takes longer than 30 s. The request is dropped mid-body, and the whole window fails, even though WINDOW_WAIT allows 5 minutes for it.

I measured the tile under Viçosa (MG) in the file above with curl: 3,750,014 bytes in 46.9 s, at 80 KB/s. Clicking again sometimes works, when the server is faster. In the CLI, the first read of a soil layer failed this way in 3 of 4 runs.

Steps to reproduce
  1. In rustgis-cli run --yes, run:

    layer.addUrl={"url": "https://ceph.opengeohub.org/arco/soildb/soildbpc_ccwpml_m_30m_bg0..30cm_20000101_20051231_go01_epsg4326_v1.2.0_srl04.tif", "bbox": [-43.01, -20.85, -42.75, -20.64], "zoom": false}
    
  2. When the server gives less than about 130 KB/s, it fails with timeout: global after about 30 s.

  3. curl -r 225825659704-225829409717 -o NUL -w "%{time_total}" <url> reads that one tile, and takes longer than 30 s.

Expected behavior

A tile that is still arriving isn't dropped. The window as a whole keeps its 5-minute wait. A server that stops sending is still given up on.

Choices in this proposal
  • A second agent for long reads: COG range reads of more than 1 MB use a second shared agent, long_read_agent() next to agent(), whose global timeout is the window's 5-minute wait (WINDOW_WAIT). Reads of 1 MB or less, and map tiles, keep the 30-second agent.
  • Why an agent and not a per-request setting: the comment on config_builder says a request's own settings make the agent set up TLS again. A second shared agent keeps its own pool, like download_agent.
  • A stalled server: the 5-minute ceiling and the connect timeout stay, so a server that never answers still fails. An idle timeout (no bytes for N seconds) would be better, but ureq 3 has none.
  • Test: a cog.rs test serves a 2 MB tile from a local server that sends it over 35 s. It checks that the window reads, and that a 20 KB tile from a stalled server still fails at 30 s.
How are you running RustGIS?

Desktop app (Windows installer), and rustgis-cli

Version

0.7.0; the 30 s agent is the same on main (e4cc3aed)

Operating system, browser and graphics

Windows 11, from Brazil

Sample data

OpenLandMap-soildb (OpenGeoHub, CC BY 4.0), any layer in https://stac.opengeohub.org/v1/cat/openlandmap, such as the file above.

Screenshots or logs

No response

Dominant language
Rust
Stars
1.9k
Forks
268
Avg merge
11h 1m
Merged PRs (30d)
942

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from opengeos/RustGIS

All issues in opengeos/RustGIS

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.