Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Tracking: large data (layers over 2 million features, slow hillshade and GeoTIFF writing)

Open
  • Is something wrong?
#1,111 8 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
rust, web-dev
Domain
backend, data, databases, performance

Research direction

Start by reading the decision section and the linked issues #1105 and #1108; the proposed work spans crates/data/src/source/mod.rs, crates/analysis/src/rasterops.rs, and crates/data/src/raster/cogwrite.rs. The issue asks contributors to first choose between streaming tools and a memory budget, then implement a broad performance plan. Done means the chosen approach and its steps are completed, with the stated correctness and performance goals met.

Written by the indexing model from the issue text.

Description

enhancement

On real-sized data RustGIS either refuses or is much slower than it needs to be. Measured on one US state (3,212,306 building footprints, 4,562 census block groups, a 9667 × 11947 DEM; rustgis-cli at d58cb8a, i7-6700, single runs, ±15%), with the time an established tool takes on the same files in brackets. This issue tracks fixing that, and has one decision to make first.

What's wrong now

  • Layers over 2,000,000 features can't be exported, reprojected or used in tools. MAX_LIMIT (crates/data/src/source/mod.rs) refuses them: all 3.2M buildings can't be converted, and #1108 is worse, a silent half sample. It's a count, not memory: it refuses 3.2M small polygons that fit in about 1.2 GB, but doesn't stop 2M large ones from exhausting memory.
  • Exports hold the whole layer, sometimes twice. Converting 1.79M polygons to FlatGeobuf takes 22.7 s and 1.2 GB (8.3 s, 0.26 GB). Reprojection clones the Dataset, and the FlatGeobuf writer builds the whole file in a Vec<u8>.
  • Hillshade is slow. It takes 59 s (7 s). gradient() in crates/analysis/src/rasterops.rs looks up the coordinate system for every cell, 115M times here, and runs five trig functions per cell. Its output is written as Float32 although the values are whole numbers 0–255.
  • Writing GeoTIFFs is slow. The COG export takes 43 s (16–22 s), nearly all of it Deflate at level 6 through miniz_oxide, about 18 MB/s (crates/data/src/raster/cogwrite.rs).
  • Summarize Within compares every point with every polygon (#1105): over 28 minutes for 1.79M points (15 s).

Needs a decision: tools on layers bigger than memory

Steps 2 and 3 below happen either way: exports stream, and the count becomes a memory budget. What should tools do past the budget?

  • A. Streaming tools (large). Per-feature tools (reproject, buffer, calculate geometry) work a chunk at a time. Tools that compare two layers (count points, spatial join, clip, select by location) index the smaller layer and stream the larger. Outputs go to temporary files added as streamed layers. That means tools on any size, but every tool gets a second execution path, and temporary-file outputs touch undo, project saving and the web app, which has no temporary files.
  • B. Stay in memory (small to very large). Tools keep working on a whole layer under the budget, and past it the answer is "clip or filter first". There is one execution path, but a ceiling, lower in the web app. Small changes (boxing the common one-ring polygon, reserving from the known count) might take a feature from about 355 to 250 bytes. A flat, columnar geometry model would roughly halve memory, but touches every crate.

Which fits where RustGIS is heading?

The steps also assume these defaults. Say so if any is wrong:

  • hillshade writes Byte;
  • threads on the desktop only, with the web app single-threaded as the renderer already is (threads_for);
  • a budget of 4 GiB on the desktop and 1 GiB in the web app, adjustable;
  • COGs use the floating-point predictor (TIFF Predictor 3), which GDAL, QGIS and the tiff crate read.

Plan

1. Correctness (small, independent)
  • Refuse instead of sampling (#1108): an incomplete read in source::materialize becomes an error, checked from the known count before reading.
  • Index Summarize Within (#1105): the polygons in ExtentIndex, each point tested against its candidates only.
2. Stream exports and reprojection (medium)
  • visit_chunks for GeoPackage, FlatGeobuf and GeoParquet. Shapefile already has it (#681).
  • A writer per format that takes chunks, and share.exportLayer on a streamed layer reading chunk → reproject → writer. Memory goes from 1.2 GB to about 0.3 GB, and the limit no longer applies to exports.
  • FlatGeobuf in the web app: its index needs a temporary file, so write it unindexed there.
3. A memory budget instead of MAX_LIMIT (small, after 2)
  • load_all counts bytes as it reads chunks and refuses past the budget, with try_reserve for the large vectors.
4. Terrain kernels (small, independent)
  • Hillshade, slope and aspect as row passes: constants computed once, three row slices walked with windows(3) and zip, with no-data and edges on the existing path. A standalone test gave identical output on all 115M cells, 21 s → 4 s on one thread, in the web app too.
  • Hillshade writes Byte.
5. GeoTIFF writing (small to medium, independent)
  • Deflate level 1 with Predictor 3 in cogwrite.rs: 157 MB/s and 25% smaller on the DEM's tiles; the COG export goes to about 12 s.
  • The untiled export through the in-house writer: tiff 0.11.3 can't write Predictor 3, and its speed today depends on zip turning on zlib-rs.
6. Threads on the desktop (medium, after 4 and 5)
  • Raster kernels in row bands, COG tiles compressed in parallel: about 3× on 4 cores.
7. Tools past the budget (large or skipped, after the decision)
  • A: streaming tools, Summarize Within first. B: the small memory savings.

Related

#1105 and #1108 are filed separately because they're bugs whatever is decided here. Step 2 builds on #681's chunked reading. #1065 (layers read in place are read twice on every render) is about drawing, not tools, so it stays separate.

Dominant language
Rust
Stars
1.9k
Forks
268
Avg merge
3h 37m
Merged PRs (30d)
818

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from opengeos/RustGIS

All issues in opengeos/RustGIS

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.