[Feature]: read a window of a GeoTIFF in strips in place (CBERS-4A WPM from the Brazil Data Cube)
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
Research direction
Start at cog::layout in crates/data/src/raster/cog.rs, where the Parse::NotTiled("a GeoTIFF in strips is read whole") refusal is produced, and read how the existing tiled Layout (tile_w, tile_h, offsets/byte counts) is consumed by the window, stretch and cancel paths. Then look at header_read in crates/tiles/src/cog.rs to see how ranges are requested and where MAX_HEADER × 64 and MAX_BYTES are enforced. Reproduce with the CBERS-4 PAN5M file, which shows the strips refusal without the header problem. Done looks like: a stripped GeoTIFF with a window bbox reads only the needed strips by merged range requests, and oversized RowsPerStrip still fails with the clear message from the proposal.
Written by the indexing model from the issue text.
Description
Before you start
- I searched the existing issues and did not find a duplicate. #2569 covers a tiled GeoTIFF whose IFD sits past 2 GiB; this one is about a GeoTIFF in strips, which is refused even once #2569 is fixed.
- This is a feature request.
What happened?
The Brazil Data Cube (INPE) serves its CBERS-4A WPM, CBERS-4 PAN5M/PAN10M/MUX and Amazonia-1 scenes as plain GeoTIFFs: DEFLATE strips of one row, with no tiles and no overviews. A WPM panchromatic scene is 57,470 × 58,936 cells at 2 m, Int16, 2.5 GB. GDAL reads a 26 × 21 km window of it through /vsicurl/, fetching only the rows it needs (572 MB, 22 % of the file), but slowly: the server gives about 1 MB/s per connection. The same rows fetched by range in 32 parallel requests of 4 MB took 38–52 s per scene (about 15 MB/s), and GDAL then cut the window from that local copy in 20 s.
RustGIS 0.7.0 refuses it:
layer.addUrl {"url": "https://data.inpe.br/bdc/data/CB4A-WPM-L4-DN/2026_07/CBERS_4A_WPM_RAW_2026_07_02.12_33_21_ETC2/198_139_0/4_BC_UTM_WGS84/CBERS_4A_WPM_20260702_198_139_L4_BAND0.tif",
"bbox": [-43.003, -20.838, -42.754, -20.655]}
→ the GeoTIFF's header is too large or too scattered to read in place
There are two causes in a row:
- The IFD and the two strip tables are written at the end of the file (offset 2,557,552,938 of 2,558,025,119, 472 KB in one piece).
header_readincrates/tiles/src/cog.rsrefuses any read that starts pastMAX_HEADER × 64(2 GiB). That is #2569. - Past that check,
cog::layoutanswersParse::NotTiled("a GeoTIFF in strips is read whole")(crates/data/src/raster/cog.rs). The only way left is to download 2.5 GB per scene and band to cut out a few km².
A CBERS-4 PAN5M scene of the same cube (216 MB, IFD at 215 MB, so #2569 does not apply) shows cause 2 alone:
layer.addUrl {"url": "https://data.inpe.br/bdc/data/cbers4/2024_04/CBERS_4_PAN5M_DRD_2024_04_25.12_39_30_CB11/151_123_0/4_BC_UTM_WGS84/CBERS_4_PAN5M_20240425_151_123_L4_BAND1.tif",
"bbox": [-43.003, -20.838, -42.754, -20.655]}
→ not a cloud-optimised GeoTIFF: a GeoTIFF in strips is read whole
The same applies to cube.reduce / stac.addCube on these collections (checked: cube.reduce on CB4A-WPM-L4-DN-1, asset BAND0, gives the header error above) (CB4A-WPM-L4-DN-1, CB4-PAN5M-L4-DN-1, CB4-MUX-L4-DN-1, AMZ1-WFI-L4-DN-1…), which read through the same window code.
Steps to reproduce
stac.search {"url": "https://data.inpe.br/bdc/stac/v1/", "collections": ["CB4A-WPM-L4-DN-1"], "bbox": [-43.003, -20.838, -42.754, -20.655], "datetime": "2026-04-01/2026-07-31"}finds 7 scenes.layer.addUrlwith theBAND0address of any of them and thatbbox.- The error above. With GDAL:
gdal_translate -projwin 707700 7714860 734110 7694050 /vsicurl/<address> out.tifwrites the 13,205 × 10,405 window.
Proposal
Treat a strip as a tile one image wide: rows r0..r1 of the window map to strips r0 / RowsPerStrip ..= r1 / RowsPerStrip, each read by range from StripOffsets / StripByteCounts, decoded, and cropped to the window's columns. Consecutive strips are merged into one range request (they are contiguous in these files), so a 10,000-row window is a handful of requests.
Choices in this proposal
- Strips become tiles of
width × RowsPerStripinsidecog::layout(aLayoutwithtile_w = width,tile_h = rows_per_strip), rather than a separate reader: the window, stretch and cancel paths already work on tiles, and they stay one code path. - Merge consecutive strips into requests of a few MB, several in parallel: strips of one row are far too small to fetch one by one, and this server caps each connection at about 1 MB/s, so one large request is slow too (8 parallel requests is a reasonable default; 32 gave 15 MB/s here).
- No overviews: cap the window by the cells read, as today (64 million values); with
cellSizecoarser than the file, read every n-th strip (decimation by rows), which is what GDAL does and keeps a coarse preview cheap. - Header at the end of the file is left to #2569's fix; this issue only lifts the strips refusal.
- Strips with
RowsPerStriplarge enough that one strip passesMAX_BYTESstay refused with a clear message ("strips of N rows are too large to read in place").
Environment
RustGIS 0.7.0 (Windows 11, x64), MCP rustgis-cli; GDAL 3.13.3 for the comparison.
- Dominant language
- Rust
- Stars
- 1.9k
- Forks
- 268
- Avg merge
- 11h 1m
- Merged PRs (30d)
- 942
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from opengeos/RustGIS
-
[Bug]: layer.move to the place a layer already has fails and stops a script; with several layers it leaves an empty undo stepPossibly taken @jgregoriorincon claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
opengeos/RustGIS#2621 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 61/100
opengeos/RustGIS#1446 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 52/100
Maintainers usually reply within 1 day
All issues in opengeos/RustGIS
Similar issues
-
postgresql-rules / queue-rules: implicit KEY SHARE before a parent key update, and job self-overlapOpen
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
jonathanong/no-mistakes#3879 ·
Maintainers usually reply within 1 day
-
area:index v0.35.0
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
nautechsystems/nautilus_trader#5741 ·
Maintainers usually reply within 1 day
-
RunPlugin derives Hash over all fields while PartialEq compares only location and configurationOpen
Difficulty 1/5 Under an hour Newbie friendliness 86/100
zellij-org/zellij#5969 ·
Maintainers usually reply within 1 day
-
eth_get_proof reports another account's fields for an absent accountPossibly taken @Indian-boult claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day