Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Materialize typed Parquet datasets with provenance

Aperta
  • Qualcosa non va?
#14 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Tranquilla
Stack tecnologico
kotlin
Ambito
data-engineering, documentation

Direzione di ricerca

Esamina prima i blockers #10 e #13, poi traccia lo stream validato/ricomposto e gli entry point della schema registry che forniscono. Definisci il confine di DatasetSink e l’implementazione esclusivamente Parquet in base ai criteri di accettazione; completato significa dataset tipizzati deterministici, output di provenienza e manifest, riepiloghi della qualità con limiti, quarantena sicura degli input non validi e nessun connettore di database.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area: analysis P1

Renamed by #23. This issue was written while the project was called Android Data
Collector. Its terminology has been updated to the Particeps identity now in force. The
old spellings are rejected inputs, not an earlier dialect.

Summary

Materialize the validated event stream from particeps-analysis as deterministic, strictly typed Parquet datasets with complete source provenance.

This issue implements Parquet only. It defines a narrow DatasetSink extension point for later work but does not add Postgres, D1, dashboards, or any other database connector.

Requirements

Strict schema materialization
  • Consume only the validated/reassembled stream from #13.
  • Resolve every collector, payload type, field, type, unit, and nullability rule through the registry in #10.
  • Convert Protocol v1 decimal-string 64-bit values to lossless Arrow/Parquet integer types after strict bounds checking.
  • Parse enumerations, booleans, integers, fixed binary, and strings according to the declared schema; do not infer types from observed values.
  • Reject/quarantine unknown collector or payload schemas, undeclared fields, invalid enums, lossy values, unit mismatches, and nullability violations. Do not emit a generic map/JSON fallback column.
  • Preserve wall time, continuous/monotonic time, time-zone context, boot-session identity, sequence numbers, and producer metadata at full declared precision.
Dataset layout and provenance
  • Partition safely and deterministically by experiment, configuration, collector, schema, and payload type, with path-safe encodings that cannot traverse the output root.
  • Within files, use stable ordering and a documented row-group/file sizing policy.
  • Make each row or row group traceable to:
    • source ciphertext SHA-256;
    • bundle ID;
    • configuration digest;
    • parser/catalog version;
    • source object/path;
    • identical-duplicate source set where applicable.
  • Keep sensitive identifiers inside the protected dataset and out of filenames unless the documented data-handling policy explicitly permits them.
  • Emit a dataset manifest containing schema fingerprints, partitions, row/file counts, source inventory digest, tool versions, conflicts/quarantines, and reproducibility metadata.
  • Define a typed DatasetSink interface around validated records, but implement only the Parquet sink in this issue.
Quality summaries

Produce offline, evidence-bounded summaries for:

  • sequence gaps, overlaps, identical duplicates, and conflicts;
  • boot sessions and incomparable monotonic intervals;
  • wall-clock and time-zone changes;
  • requested versus achieved sampling rates where the configuration and data support the calculation;
  • collector start/stop/access/quota states;
  • intervention/survey occurrence lifecycle.

Do not infer participant awareness, notification visibility, travel, presence, orientation, activity, or other behavior the recorded events cannot prove.

Acceptance criteria

  • Every shipped registry schema has deterministic Arrow/Parquet schema fixtures.
  • Unknown schemas or invalid typed values fail closed without a generic fallback partition.
  • Full-range supported 64-bit sequence/time values round-trip without floating-point conversion or precision loss.
  • Identical validated inputs, catalog version, and options produce identical logical schemas, manifests, partition paths, ordering, and quality summaries.
  • Any output row can be traced to its source ciphertext, bundle, configuration, parser/catalog version, and source object.
  • Local and R2-derived validated streams produce equivalent Parquet datasets.
  • Conflicts and quarantined bundles are reported and never silently materialized as authoritative rows.
  • The shipped package has no database client or D1/Postgres-specific implementation.

Documentation

Document Parquet schemas and partitioning, provenance columns/sidecars, manifest format, quality-metric definitions and limits, secure output handling, reproducibility workflow, and the future DatasetSink contract.

Dependencies

Blocked by #10 (schema registry) and #13 (validated reassembly stream).

Lingua principale
Kotlin
Stelle
0
Fork
0
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di JacobLinCool/particeps

Tutte le issue di JacobLinCool/particeps

Issue simili

Altre issue su Kotlin

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.