Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Data compression

Aperta
  • Qualcosa non va?
#8 7 commenti 1 reazione 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
20/100
Tipo di issue
Funzionalità
Chiarezza
Da chiarire
Stato di attività
Ferma
Stack tecnologico
python
Ambito
data, performance

Direzione di ricerca

Iniziate leggendo la discussione sulla compressione TRX e le issue citate dipy/dipy#2229 e nipy/nibabel#942. Confrontate le opzioni elencate, tra cui l’archiviazione in float16, il downsampling, la compressione delle directory in file ZIP e le strategie interne al formato; il lavoro sarà completo solo quando sarà concordata una direzione precisa per la compressione, poiché questa issue non definisce un obiettivo di implementazione o di test.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Given recent comments in dipy/dipy#2229, and further back in nipy/nibabel#942, I think it is worth dedicating a discussion thread to the issue of data compression.

I think it is fair to say that this particular format proposal has prioritised extensibility and simplicity over compression. But for some users / developers the latter may be the higher priority. So here we can discuss whether the gains in this department from the current TRX proposal are adequate, whether it is possible to bootstrap any kind of explicit data compression into this format proposal, or whether a drastically different proposal would be necessary in order to satisfy such demands.

Please amend or correct as necessary, and comment as you see fit.

Where compression may be possible in TRX:

  1. Use of float16's for vertex locations -> halves storage over float32
  2. No need for delimiters -> one less vertex per streamline compared to delimiter-based formats, though explicitly storing offsets reduces that gain slightly
  3. Compatible with downsampling, i.e. storing less vertices per streamline than what was used in generation of the streamline; this can be a naive integer factor reduction in vertex count, or based on estimation of error accumulation (e.g. https://www.sciencedirect.com/science/article/pii/S1053811914010635); in either case for algorithms with small step size this can be the primary source of storage reduction
  4. Zipping of full directory structure (albeit depending on implementation this may impact IO substantially), though yields with this tend to be minimal due to the nature of the data
  5. Support for more dedicated compression strategies as in-place alternatives for vertex position (or indeed DPP / DPS / DPG) data within TRX.

Point 5 may be interesting to explore: there's been a number of manuscripts published explicitly on storage reduction of streamline vertex information, so simply having the option to utilise within TRX one such format for vertex position information would theoretically be possible. It would make the technical specification considerably more complex, but technically one could defer to external specifications for datasets doing such. I'm not a fan of this personally, I'm simply stating that it's a prospect.

In the converse case, i.e. a format from the ground up that would prioritise compression: A couple of these already exist, but don't have the extensibility of TRX. Trying to extend the capabilities of compressed streamline data storage could look something like bootstrapping compression into the .trk format, or it could be something entirely novel. It probably falls upon someone who prioritises such to propose something that still has a chance of satisfying the desires of those prioritising extensibility. But this is likely one of multiple dimensions of difference that caused the original discussion in nipy/nibabel#942 to go slightly astray, so warrants greater precision here.

Lingua principale
Python
Stelle
27
Fork
20
Merge medio
2g 25m
PR unite (30g)
1

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di tee-ar-ex/trx-python

Tutte le issue di tee-ar-ex/trx-python

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.