maplibre/martin

Store hashing algorithm in the metadata table

Open

#1,086 opened on Dec 19, 2023

 (2 comments) (0 reactions) (0 assignees)Rust (377 forks)auto 404
good first issuehelp wantedmbtilesrust

Repository metrics

Stars
 (3,761 stars)
PR merge metrics
 (PR metrics pending)

Description

mbtiles and martin-cp standardize how a tile data is hashed in the mbtiles. tilelive-copy has been generating normalized mbtiles schema with MD5 hashes of the tile data as the table foreign key. So it was possible to validate the content of the tile by re-computing MD5 hash.

tippecanoe (cc @bdon) has used a faster fnv1a checksum algorithm, so clearly there is need for more than one algo.

Proposal

  • Add hash_algorithm metadata value with the possible values md5, fnv1a, ...
  • If the value is not there, check if agg_tiles_hash metadata value is present, and use its length to decide algorithm
  • If not, and mbtiles uses normalized or flat-with-hash schema, use the size of the hashing keys to decide the algorithm
  • If the hashing algorithm is detected, add it to the metadata, possibly after validating that all hashes match their data

Multiple Algorithms Consideration

MBTiles use hashes for the tile_data column and for the agg_tiles_hash metadata field. In theory these could use different algorithms for performance and security(?) reasons. If so, we could either have two metadata fields (hash_algorithm and agg_hash_algorithm), or better yet, rename agg_tiles_hash into agg_tiles_hash_md5 (or other algo). This keeps the algorithm and its value together, and is fairly easy to detect at runtime.

Contributor guide