[Feature]: read a window of a GeoTIFF in strips in place (CBERS-4A WPM from the Brazil Data Cube)
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 25/100
Direzione di ricerca
Start at cog::layout in crates/data/src/raster/cog.rs, where the Parse::NotTiled("a GeoTIFF in strips is read whole") refusal is produced, and read how the existing tiled Layout (tile_w, tile_h, offsets/byte counts) is consumed by the window, stretch and cancel paths. Then look at header_read in crates/tiles/src/cog.rs to see how ranges are requested and where MAX_HEADER × 64 and MAX_BYTES are enforced. Reproduce with the CBERS-4 PAN5M file, which shows the strips refusal without the header problem. Done looks like: a stripped GeoTIFF with a window bbox reads only the needed strips by merged range requests, and oversized RowsPerStrip still fails with the clear message from the proposal.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Before you start
- I searched the existing issues and did not find a duplicate. #2569 covers a tiled GeoTIFF whose IFD sits past 2 GiB; this one is about a GeoTIFF in strips, which is refused even once #2569 is fixed.
- This is a feature request.
What happened?
The Brazil Data Cube (INPE) serves its CBERS-4A WPM, CBERS-4 PAN5M/PAN10M/MUX and Amazonia-1 scenes as plain GeoTIFFs: DEFLATE strips of one row, with no tiles and no overviews. A WPM panchromatic scene is 57,470 × 58,936 cells at 2 m, Int16, 2.5 GB. GDAL reads a 26 × 21 km window of it through /vsicurl/, fetching only the rows it needs (572 MB, 22 % of the file), but slowly: the server gives about 1 MB/s per connection. The same rows fetched by range in 32 parallel requests of 4 MB took 38–52 s per scene (about 15 MB/s), and GDAL then cut the window from that local copy in 20 s.
RustGIS 0.7.0 refuses it:
layer.addUrl {"url": "https://data.inpe.br/bdc/data/CB4A-WPM-L4-DN/2026_07/CBERS_4A_WPM_RAW_2026_07_02.12_33_21_ETC2/198_139_0/4_BC_UTM_WGS84/CBERS_4A_WPM_20260702_198_139_L4_BAND0.tif",
"bbox": [-43.003, -20.838, -42.754, -20.655]}
→ the GeoTIFF's header is too large or too scattered to read in place
There are two causes in a row:
- The IFD and the two strip tables are written at the end of the file (offset 2,557,552,938 of 2,558,025,119, 472 KB in one piece).
header_readincrates/tiles/src/cog.rsrefuses any read that starts pastMAX_HEADER × 64(2 GiB). That is #2569. - Past that check,
cog::layoutanswersParse::NotTiled("a GeoTIFF in strips is read whole")(crates/data/src/raster/cog.rs). The only way left is to download 2.5 GB per scene and band to cut out a few km².
A CBERS-4 PAN5M scene of the same cube (216 MB, IFD at 215 MB, so #2569 does not apply) shows cause 2 alone:
layer.addUrl {"url": "https://data.inpe.br/bdc/data/cbers4/2024_04/CBERS_4_PAN5M_DRD_2024_04_25.12_39_30_CB11/151_123_0/4_BC_UTM_WGS84/CBERS_4_PAN5M_20240425_151_123_L4_BAND1.tif",
"bbox": [-43.003, -20.838, -42.754, -20.655]}
→ not a cloud-optimised GeoTIFF: a GeoTIFF in strips is read whole
The same applies to cube.reduce / stac.addCube on these collections (checked: cube.reduce on CB4A-WPM-L4-DN-1, asset BAND0, gives the header error above) (CB4A-WPM-L4-DN-1, CB4-PAN5M-L4-DN-1, CB4-MUX-L4-DN-1, AMZ1-WFI-L4-DN-1…), which read through the same window code.
Steps to reproduce
stac.search {"url": "https://data.inpe.br/bdc/stac/v1/", "collections": ["CB4A-WPM-L4-DN-1"], "bbox": [-43.003, -20.838, -42.754, -20.655], "datetime": "2026-04-01/2026-07-31"}finds 7 scenes.layer.addUrlwith theBAND0address of any of them and thatbbox.- The error above. With GDAL:
gdal_translate -projwin 707700 7714860 734110 7694050 /vsicurl/<address> out.tifwrites the 13,205 × 10,405 window.
Proposal
Treat a strip as a tile one image wide: rows r0..r1 of the window map to strips r0 / RowsPerStrip ..= r1 / RowsPerStrip, each read by range from StripOffsets / StripByteCounts, decoded, and cropped to the window's columns. Consecutive strips are merged into one range request (they are contiguous in these files), so a 10,000-row window is a handful of requests.
Choices in this proposal
- Strips become tiles of
width × RowsPerStripinsidecog::layout(aLayoutwithtile_w = width,tile_h = rows_per_strip), rather than a separate reader: the window, stretch and cancel paths already work on tiles, and they stay one code path. - Merge consecutive strips into requests of a few MB, several in parallel: strips of one row are far too small to fetch one by one, and this server caps each connection at about 1 MB/s, so one large request is slow too (8 parallel requests is a reasonable default; 32 gave 15 MB/s here).
- No overviews: cap the window by the cells read, as today (64 million values); with
cellSizecoarser than the file, read every n-th strip (decimation by rows), which is what GDAL does and keeps a coarse preview cheap. - Header at the end of the file is left to #2569's fix; this issue only lifts the strips refusal.
- Strips with
RowsPerStriplarge enough that one strip passesMAX_BYTESstay refused with a clear message ("strips of N rows are too large to read in place").
Environment
RustGIS 0.7.0 (Windows 11, x64), MCP rustgis-cli; GDAL 3.13.3 for the comparison.
- Lingua principale
- Rust
- Stelle
- 1.9k
- Fork
- 268
- Merge medio
- 11h 1m
- PR unite (30g)
- 942
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di opengeos/RustGIS
-
[Bug]: layer.move to the place a layer already has fails and stops a script; with several layers it leaves an empty undo stepForse già presa @jgregoriorincon l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
opengeos/RustGIS#2621 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 65/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 85/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 61/100
opengeos/RustGIS#1446 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 3/5 1-2 giorni Idoneità per principianti 52/100
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di opengeos/RustGIS
Issue simili
-
文件夹重大BUG:不支持重命名Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 64/100
I maintainer di solito rispondono entro 1 giorno
-
postgresql-rules / queue-rules: implicit KEY SHARE before a parent key update, and job self-overlapAperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
jonathanong/no-mistakes#3879 ·
I maintainer di solito rispondono entro 1 giorno
-
area:index v0.35.0
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
I maintainer di solito rispondono entro 1 giorno
-
needs triage
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
pbakaus/impeccable#1653 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 80/100
nautechsystems/nautilus_trader#5741 ·
I maintainer di solito rispondono entro 1 giorno