Real-project storage and compute: streamed datasets and bounded GPU execution

Aperta
  • Qualcosa non va?
#1,241 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Da chiarire
Stato di attività
Tranquilla
Stack tecnologico
huggingface
Ambito
ai-infra-agents, cloud, data-engineering

Direzione di ricerca

Non sono stati identificati file del repository, test o punti di ingresso. Inizia esaminando i riferimenti a hf-mount, Datasets streaming e ZeroGPU, quindi analizza come questo repository potrebbe esporre strumenti per gli agenti. Limita la proposta a un’integrazione e definisci test per il relativo comportamento di archiviazione, dataset o calcolo GPU prima dell’implementazione.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

for storage and compute: we need to make better use of the huggingface infrastructure. e.g., hf-mount (https://github.com/huggingface/hf-mount) could be a great way of handling very large datasets in resource constrained environments.

stream (https://huggingface.co/docs/datasets/en/stream) is also a nice way of handling large datasets.

we could also use zerogpu (https://huggingface.co/docs/hub/en/spaces-zerogpu) for GPU compute

it could be useful to create (and test) tools that agents can then use to more effectively research

October 8 backlog reconciliation

Retained as a concrete capability gap rather than closed by age. PR #1472's CPU canary and safer execution environment do not implement streamed/mounted large datasets or GPU execution. Before selecting an integration, use a blocked real project to establish required fields, data volume, memory/compute limits and access terms; then demonstrate a real load/compute/artifact path with bounded cost and reproducible provenance. Discovery/recovery triggers and scientific outcome checks belong to #1139; this issue owns any new storage/compute capability. The originally named services are candidates, not evidence of current suitability or a mandate to add all three.

Measured repository storage and startup cost — October 9

At commit 8cfc7e03740b1250736f6bd25ac2bb1fe1a48179, git ls-tree -r -l reports 267,658 tracked files and 11,855,195,572 uncompressed blob bytes. Project artifacts account for 8,682,422,267 bytes across 91,547 files; docs account for 2,233,050,304 bytes; state has 169,130 files and 477,792,297 bytes. Local packed history is 8.62 GiB. These are Git snapshot measurements, not process RAM or a claim that every byte is redundant.

The grounding-fix CI run spent 268 seconds in checkout, 520 seconds in offline unit tests and 215 seconds in real-call tests. The repair trial spent 307 seconds in checkout. Startup/file-enumeration cost is a material throughput issue alongside model calls.

  • Design a retention/storage migration for large source PDFs, duplicated rendered PDFs and generated site assets, with immutable provenance, verified retrieval and backups before removing tracked copies. Historical root-level downloaded datasets also need ownership/provenance review; do not silently delete scientific inputs.
  • Measure selective/partial checkout and indexed state access on actual CI and advancement workloads. Preserve required fixture coverage and real project artifact access; a smaller checkout that skips required tests or source evidence is not acceptance.
  • Demonstrate lower end-to-end startup time on the same workload after migration. Keep code, compact manifests and active state in Git while evaluating appropriate artifact storage for bulk data; do not choose a service from availability alone.
Lingua principale
Python
Stelle
4
Fork
2
Merge medio
36m
PR unite (30g)
109

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di ContextLab/llmXive

Tutte le issue di ContextLab/llmXive

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.