Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Manifest Format

Aperta
  • Qualcosa non va?
#321 6 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
20/100
Tipo di issue
Funzionalità
Chiarezza
Da chiarire
Stato di attività
Ferma
Stack tecnologico
rust
Ambito
cli, cryptography, security

Direzione di ricerca

Non sono identificati file di implementazione, test o punti di ingresso. Inizia esaminando i requisiti del manifest e le opzioni proposte per CBOR, MessagePack, BLAKE3, flat/tree e release-layout; il design è completo quando queste scelte sono state risolte in una specifica del manifest e in un piano di implementazione definiti.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

design

Many features are gated on the basic design of the Intermodal manifest. So let's get started on it right away.

Desiderata

  • Allow integrity checking. Given a manifest, an accompanying release can be checked for integrity using the manifest. This will require including secure hashes of accompanying files in the manifest.

  • Hashing the manifest should give a secure hash that uniquely identifies the contents of the release.

  • Multi-level manifest. A lower-level manifest should commit to the contents of a release. A higher -level manifest should commit to both the lower level manifest, as well as any files containing signatures over the lower level manifest. It would be nice to only have one manifest, but since data can't self-sign or contain hashes to itself, it seems necessary to have at least a two-level manifest so that we can produce a hash that uniquely identifies a collection of files, as well as signatures and other commitments to that collection of files.

    I'm thinking about calling the lower-level manifest the "content manifest" and the lower-level "bundle manifest". I'm definitely open to naming suggestions though. Other ideas are "file manifest" and "root manifest".

Why not use BitTorrent metainfo?

  • BitTorrent v1 uses SHA1, which is insecure.

  • BitTorrent v2 uses a custom tree hash that is vulnerable to attack if the length of the content is not included.

  • Bencoding is not a particularly popular encoding format.

Why not use the web packaging format?

The web bundle format is a single-file format, so it would be impossible to use natively with BitTorrent, which is an important transport.

Out of Scope

To keep things simple, it would be a good idea to limit the scope of the initial manifest design as much as possible. Things that we should consider for the design, but not worry about the details:

  • Metadata. Structured metadata can be included in a file that the content manifest commits to.

  • Signatures, timestamps, and related functionality. A two-level manifest leaves open the ability to include files that are signatures over the hash of the content manifest, which are committed to by the bundle manifest.

In scope

  • Manifest format. I'm thinking either CBOR or messagepack. They are both lightweight, binary, schemaless formats with an object model that is similar to JSON. The web packaging format uses CBOR, so that's what I'm leaning towards. Keybase's saltpack, however, uses messagepack, so that's a contender too.

  • The hash function. Since manifests will have to include secure hashes, we should pick a hash function. I'm leaning towards BLAKE3, although someone could probably talk me out of it. BLAKE3 is extremely fast, supports random access and streaming verification, and has a strong rust implementation, all of which are nice features. On the downside, it is very new, and uses a reduced strength construction. However, that reduced strength is argued to not be vulnerable now or in the future.

  • Whether the manifest should be flat or a tree. Nested will be more compact when there are long directory names with many entires, but is more complex. Nested doesn't explicitly encode path separators, which I think is a bonus.

    flat: {"foo/bar": "BAR_HASH", "foo/baz": "BAZ_HASH"}
tree: {foo: {bar: "BAR_HASH", baz: "BAZ_HASH"}}

    BitTorrent V2 uses a tree, so that's what I'm leaning towards.

  • Where to put the manifest in a release. Since it seems likely that we'll eventually want multiple files, I'm thinking that putting everything into a subdirectory is a good idea, either imdl/ or intermodal/.

    For example, if we go with CBOR, the structure could be:

    intermodal/root.cbor     # root manifest
intermodal/content.cbor  # content manifest
intermodal/signatures    # signatures over hash of content.manifest
intermodal/timestamp.ots # open timestamps timestamp
intermodal/metadata.cbor # metadata
intermodal/README.txt    # human readable info about intermodal

Postscript

A very weird but nonetheless interesting choice of format would be FIDL, Fuchsia's IPC system:

  • Modular compiler and language bindings.
  • Message encoding is canonical — there is exactly one encoding for a given message.
  • Supports both fixed-size messages where space is important, and extensible tables and unions where schema evolution is important.
  • Defined to be little endian and uses natural alignment, so no portability issues.
  • Zero copy and zero parse, by storing variable sized members out-of-line. Can be memory mapped with LMDB or mmap(2) for extremely fast access.
  • Certain types can be introspected without access to a schema, namely tables and unions.
  • Could make it fully introspectable where needed by including hash of schema as first element of messages.

Another less wacky choice would be flatbuffers. Flatbuffers also support zero copy and zero parse deserialization.

Misc

  • Would like to compare the encoding choices of flatbuffer and FIDL.
  • Would like to add some kind of link attribute, that would indirect through a hash.
  • Unsure of how to approach "canonical encoding". Forcing the format to output a buffer in a canonical format is inflexible. A more flexible approach would be to calculate hashes over logical traversals of the physical data, so the physical data may be in multiple forms, but hash to the same value, as long as the traversal doesn't change.
  • Flatbuffers binary format documentation
  • fleetfs flatbuffers schema
  • proc macro attribute parser
  • syn helper
Lingua principale
Rust
Stelle
662
Fork
36
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di casey/intermodal

Tutte le issue di casey/intermodal

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.