[Feature]: read a window of a GeoTIFF in strips in place (CBERS-4A WPM from the Brazil Data Cube)

Aperta
#2,592 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
rust
Ambito
data

Direzione di ricerca

Start at cog::layout in crates/data/src/raster/cog.rs, where the Parse::NotTiled("a GeoTIFF in strips is read whole") refusal is produced, and read how the existing tiled Layout (tile_w, tile_h, offsets/byte counts) is consumed by the window, stretch and cancel paths. Then look at header_read in crates/tiles/src/cog.rs to see how ranges are requested and where MAX_HEADER × 64 and MAX_BYTES are enforced. Reproduce with the CBERS-4 PAN5M file, which shows the strips refusal without the header problem. Done looks like: a stripped GeoTIFF with a window bbox reads only the needed strips by merged range requests, and oversized RowsPerStrip still fails with the clear message from the proposal.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Before you start
  • I searched the existing issues and did not find a duplicate. #2569 covers a tiled GeoTIFF whose IFD sits past 2 GiB; this one is about a GeoTIFF in strips, which is refused even once #2569 is fixed.
  • This is a feature request.
What happened?

The Brazil Data Cube (INPE) serves its CBERS-4A WPM, CBERS-4 PAN5M/PAN10M/MUX and Amazonia-1 scenes as plain GeoTIFFs: DEFLATE strips of one row, with no tiles and no overviews. A WPM panchromatic scene is 57,470 × 58,936 cells at 2 m, Int16, 2.5 GB. GDAL reads a 26 × 21 km window of it through /vsicurl/, fetching only the rows it needs (572 MB, 22 % of the file), but slowly: the server gives about 1 MB/s per connection. The same rows fetched by range in 32 parallel requests of 4 MB took 38–52 s per scene (about 15 MB/s), and GDAL then cut the window from that local copy in 20 s.

RustGIS 0.7.0 refuses it:

layer.addUrl {"url": "https://data.inpe.br/bdc/data/CB4A-WPM-L4-DN/2026_07/CBERS_4A_WPM_RAW_2026_07_02.12_33_21_ETC2/198_139_0/4_BC_UTM_WGS84/CBERS_4A_WPM_20260702_198_139_L4_BAND0.tif",
              "bbox": [-43.003, -20.838, -42.754, -20.655]}
→ the GeoTIFF's header is too large or too scattered to read in place

There are two causes in a row:

  1. The IFD and the two strip tables are written at the end of the file (offset 2,557,552,938 of 2,558,025,119, 472 KB in one piece). header_read in crates/tiles/src/cog.rs refuses any read that starts past MAX_HEADER × 64 (2 GiB). That is #2569.
  2. Past that check, cog::layout answers Parse::NotTiled("a GeoTIFF in strips is read whole") (crates/data/src/raster/cog.rs). The only way left is to download 2.5 GB per scene and band to cut out a few km².

A CBERS-4 PAN5M scene of the same cube (216 MB, IFD at 215 MB, so #2569 does not apply) shows cause 2 alone:

layer.addUrl {"url": "https://data.inpe.br/bdc/data/cbers4/2024_04/CBERS_4_PAN5M_DRD_2024_04_25.12_39_30_CB11/151_123_0/4_BC_UTM_WGS84/CBERS_4_PAN5M_20240425_151_123_L4_BAND1.tif",
              "bbox": [-43.003, -20.838, -42.754, -20.655]}
→ not a cloud-optimised GeoTIFF: a GeoTIFF in strips is read whole

The same applies to cube.reduce / stac.addCube on these collections (checked: cube.reduce on CB4A-WPM-L4-DN-1, asset BAND0, gives the header error above) (CB4A-WPM-L4-DN-1, CB4-PAN5M-L4-DN-1, CB4-MUX-L4-DN-1, AMZ1-WFI-L4-DN-1…), which read through the same window code.

Steps to reproduce
  1. stac.search {"url": "https://data.inpe.br/bdc/stac/v1/", "collections": ["CB4A-WPM-L4-DN-1"], "bbox": [-43.003, -20.838, -42.754, -20.655], "datetime": "2026-04-01/2026-07-31"} finds 7 scenes.
  2. layer.addUrl with the BAND0 address of any of them and that bbox.
  3. The error above. With GDAL: gdal_translate -projwin 707700 7714860 734110 7694050 /vsicurl/<address> out.tif writes the 13,205 × 10,405 window.
Proposal

Treat a strip as a tile one image wide: rows r0..r1 of the window map to strips r0 / RowsPerStrip ..= r1 / RowsPerStrip, each read by range from StripOffsets / StripByteCounts, decoded, and cropped to the window's columns. Consecutive strips are merged into one range request (they are contiguous in these files), so a 10,000-row window is a handful of requests.

Choices in this proposal
  • Strips become tiles of width × RowsPerStrip inside cog::layout (a Layout with tile_w = width, tile_h = rows_per_strip), rather than a separate reader: the window, stretch and cancel paths already work on tiles, and they stay one code path.
  • Merge consecutive strips into requests of a few MB, several in parallel: strips of one row are far too small to fetch one by one, and this server caps each connection at about 1 MB/s, so one large request is slow too (8 parallel requests is a reasonable default; 32 gave 15 MB/s here).
  • No overviews: cap the window by the cells read, as today (64 million values); with cellSize coarser than the file, read every n-th strip (decimation by rows), which is what GDAL does and keeps a coarse preview cheap.
  • Header at the end of the file is left to #2569's fix; this issue only lifts the strips refusal.
  • Strips with RowsPerStrip large enough that one strip passes MAX_BYTES stay refused with a clear message ("strips of N rows are too large to read in place").
Environment

RustGIS 0.7.0 (Windows 11, x64), MCP rustgis-cli; GDAL 3.13.3 for the comparison.

Lingua principale
Rust
Stelle
1.9k
Fork
268
Merge medio
11h 1m
PR unite (30g)
942

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di opengeos/RustGIS

Tutte le issue di opengeos/RustGIS

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.