[Feature]: read a window of a GeoTIFF in strips in place (CBERS-4A WPM from the Brazil Data Cube)

Open
#2,592 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
rust
Domain
data

Research direction

Start at cog::layout in crates/data/src/raster/cog.rs, where the Parse::NotTiled("a GeoTIFF in strips is read whole") refusal is produced, and read how the existing tiled Layout (tile_w, tile_h, offsets/byte counts) is consumed by the window, stretch and cancel paths. Then look at header_read in crates/tiles/src/cog.rs to see how ranges are requested and where MAX_HEADER × 64 and MAX_BYTES are enforced. Reproduce with the CBERS-4 PAN5M file, which shows the strips refusal without the header problem. Done looks like: a stripped GeoTIFF with a window bbox reads only the needed strips by merged range requests, and oversized RowsPerStrip still fails with the clear message from the proposal.

Written by the indexing model from the issue text.

Description

Before you start
  • I searched the existing issues and did not find a duplicate. #2569 covers a tiled GeoTIFF whose IFD sits past 2 GiB; this one is about a GeoTIFF in strips, which is refused even once #2569 is fixed.
  • This is a feature request.
What happened?

The Brazil Data Cube (INPE) serves its CBERS-4A WPM, CBERS-4 PAN5M/PAN10M/MUX and Amazonia-1 scenes as plain GeoTIFFs: DEFLATE strips of one row, with no tiles and no overviews. A WPM panchromatic scene is 57,470 × 58,936 cells at 2 m, Int16, 2.5 GB. GDAL reads a 26 × 21 km window of it through /vsicurl/, fetching only the rows it needs (572 MB, 22 % of the file), but slowly: the server gives about 1 MB/s per connection. The same rows fetched by range in 32 parallel requests of 4 MB took 38–52 s per scene (about 15 MB/s), and GDAL then cut the window from that local copy in 20 s.

RustGIS 0.7.0 refuses it:

layer.addUrl {"url": "https://data.inpe.br/bdc/data/CB4A-WPM-L4-DN/2026_07/CBERS_4A_WPM_RAW_2026_07_02.12_33_21_ETC2/198_139_0/4_BC_UTM_WGS84/CBERS_4A_WPM_20260702_198_139_L4_BAND0.tif",
              "bbox": [-43.003, -20.838, -42.754, -20.655]}
→ the GeoTIFF's header is too large or too scattered to read in place

There are two causes in a row:

  1. The IFD and the two strip tables are written at the end of the file (offset 2,557,552,938 of 2,558,025,119, 472 KB in one piece). header_read in crates/tiles/src/cog.rs refuses any read that starts past MAX_HEADER × 64 (2 GiB). That is #2569.
  2. Past that check, cog::layout answers Parse::NotTiled("a GeoTIFF in strips is read whole") (crates/data/src/raster/cog.rs). The only way left is to download 2.5 GB per scene and band to cut out a few km².

A CBERS-4 PAN5M scene of the same cube (216 MB, IFD at 215 MB, so #2569 does not apply) shows cause 2 alone:

layer.addUrl {"url": "https://data.inpe.br/bdc/data/cbers4/2024_04/CBERS_4_PAN5M_DRD_2024_04_25.12_39_30_CB11/151_123_0/4_BC_UTM_WGS84/CBERS_4_PAN5M_20240425_151_123_L4_BAND1.tif",
              "bbox": [-43.003, -20.838, -42.754, -20.655]}
→ not a cloud-optimised GeoTIFF: a GeoTIFF in strips is read whole

The same applies to cube.reduce / stac.addCube on these collections (checked: cube.reduce on CB4A-WPM-L4-DN-1, asset BAND0, gives the header error above) (CB4A-WPM-L4-DN-1, CB4-PAN5M-L4-DN-1, CB4-MUX-L4-DN-1, AMZ1-WFI-L4-DN-1…), which read through the same window code.

Steps to reproduce
  1. stac.search {"url": "https://data.inpe.br/bdc/stac/v1/", "collections": ["CB4A-WPM-L4-DN-1"], "bbox": [-43.003, -20.838, -42.754, -20.655], "datetime": "2026-04-01/2026-07-31"} finds 7 scenes.
  2. layer.addUrl with the BAND0 address of any of them and that bbox.
  3. The error above. With GDAL: gdal_translate -projwin 707700 7714860 734110 7694050 /vsicurl/<address> out.tif writes the 13,205 × 10,405 window.
Proposal

Treat a strip as a tile one image wide: rows r0..r1 of the window map to strips r0 / RowsPerStrip ..= r1 / RowsPerStrip, each read by range from StripOffsets / StripByteCounts, decoded, and cropped to the window's columns. Consecutive strips are merged into one range request (they are contiguous in these files), so a 10,000-row window is a handful of requests.

Choices in this proposal
  • Strips become tiles of width × RowsPerStrip inside cog::layout (a Layout with tile_w = width, tile_h = rows_per_strip), rather than a separate reader: the window, stretch and cancel paths already work on tiles, and they stay one code path.
  • Merge consecutive strips into requests of a few MB, several in parallel: strips of one row are far too small to fetch one by one, and this server caps each connection at about 1 MB/s, so one large request is slow too (8 parallel requests is a reasonable default; 32 gave 15 MB/s here).
  • No overviews: cap the window by the cells read, as today (64 million values); with cellSize coarser than the file, read every n-th strip (decimation by rows), which is what GDAL does and keeps a coarse preview cheap.
  • Header at the end of the file is left to #2569's fix; this issue only lifts the strips refusal.
  • Strips with RowsPerStrip large enough that one strip passes MAX_BYTES stay refused with a clear message ("strips of N rows are too large to read in place").
Environment

RustGIS 0.7.0 (Windows 11, x64), MCP rustgis-cli; GDAL 3.13.3 for the comparison.

Dominant language
Rust
Stars
1.9k
Forks
268
Avg merge
11h 1m
Merged PRs (30d)
942

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from opengeos/RustGIS

All issues in opengeos/RustGIS

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.