mosaicml/streaming

View on GitHub

A Data Streaming Library for Efficient Neural Network Training

Stars: 1.6k
Forks: 206
Open beginner issues: 0
Indexed issues: 77

Dominant language: Python
License: Apache-2.0
Last GitHub push: Jun 25, 2026
Latest indexed: Sep 20, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: No beginner labels indexed
PR merge metrics: No merged PRs in 30d

Beginner friendly Everything

Open All

77 open issues indexed

Data loss: cache_limit can evict canonical local-only shards when remote=None Open

Difficulty 3/5 1-2 days Newbie friendliness 72/100

mosaicml/streaming#988 · Opened Aug 17, 2026
Are there any supported patterns for on-the-fly caching? Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

mosaicml/streaming#981 · Opened May 1, 2026
Not compatible with HF transformers >=5.0.0 Open

Difficulty 2/5 1-3 hours Newbie friendliness 72/100

mosaicml/streaming#979 · 3 comments · 2 reactions · Opened Mar 12, 2026
Does MDSWriter release the GIL? Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

mosaicml/streaming#978 · Opened Mar 1, 2026
Support one to many mapping? Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

mosaicml/streaming#974 · Opened Feb 11, 2026
Support for different OCI object store authentication mechanisms Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 42/100

mosaicml/streaming#973 · Opened Feb 6, 2026
dataframe_to_mds fails on large datasets due to Worker Disk Exhaustion Open

bug

Difficulty 5/5 Over a week Newbie friendliness 35/100

mosaicml/streaming#970 · 1 comment · Opened Feb 2, 2026
Windows absolute paths with drive letters incorrectly parsed as cloud provider schemes Open

bug

Difficulty 2/5 1-3 hours Newbie friendliness 55/100

mosaicml/streaming#960 · 1 comment · Opened Nov 4, 2025
merge_index drops full directory path in merged index.json, causing broken shard references and 404s Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 68/100

mosaicml/streaming#959 · 2 comments · Opened Nov 3, 2025
Performance imporve: mmap and file handler cache Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

mosaicml/streaming#958 · 2 comments · Opened Oct 31, 2025
Load only a subset of columns Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

mosaicml/streaming#926 · 1 reaction · Opened Jun 3, 2025
Feature request: support for split/Subset-like behavior and shared local cache in StreamingDataset Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

mosaicml/streaming#916 · Opened May 15, 2025
How to sample nearby items while keeping performance? Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

mosaicml/streaming#905 · Opened Apr 22, 2025
Allow configuring max_pool_connections on boto3 s3 client Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 45/100

mosaicml/streaming#902 · Opened Apr 19, 2025
Race Condition in base.shared.get_shm_prefix (possible general SharedMemory issue) Open

bug

Difficulty 5/5 Over a week Newbie friendliness 25/100

mosaicml/streaming#901 · 8 comments · Opened Apr 14, 2025
Clean up garbage files Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mosaicml/streaming#893 · 2 comments · Opened Apr 2, 2025
StreamingDataset gives FileExistsError when called with multiprocessing Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 38/100

mosaicml/streaming#884 · 1 reaction · Opened Mar 10, 2025
Unable to get mid-epoch resumption working Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mosaicml/streaming#880 · Opened Feb 25, 2025
Streaming Dataset Simulator stops working beyond 10,000 batches Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mosaicml/streaming#877 · 1 comment · Opened Feb 12, 2025
NCCL Timeouts from Adding More Datasets + Proportion Sampling Open

bug

Difficulty 5/5 Over a week Newbie friendliness 25/100

mosaicml/streaming#876 · 6 comments · Opened Feb 11, 2025
[Bug] Incorrect local file path in HfUploader Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 48/100

mosaicml/streaming#875 · Opened Feb 9, 2025
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/mds_data/train/index.json' Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

mosaicml/streaming#873 · Opened Feb 7, 2025
exist_ok mds_kwargs not passable to dataframe_to_mds() Open

Difficulty 2/5 1-3 hours Newbie friendliness 52/100

mosaicml/streaming#871 · 1 reaction · Opened Jan 31, 2025
dataframe_to_mds fails with non-nullable ArrayType Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 52/100

mosaicml/streaming#870 · 3 comments · Opened Jan 30, 2025
Facing issue with resuming training for saved dataset state (>1 epoch) Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mosaicml/streaming#869 · 3 comments · Opened Jan 28, 2025
Facing issue with improper file renaming while multi-node training in ROCm Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mosaicml/streaming#868 · 4 comments · Opened Jan 27, 2025
how about read speed when training, just save on local. compare to use pytorch dataloader Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

mosaicml/streaming#864 · 1 comment · Opened Jan 20, 2025
Error in Streaming Dataset Decompression in Distributed Setting Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mosaicml/streaming#863 · 2 comments · 1 reaction · Opened Jan 15, 2025
Expose StreamingDataset world (or world_size and rank) as argument Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mosaicml/streaming#854 · 1 reaction · Opened Dec 19, 2024
Will cache eviction logic take previously-existing shards into account? Open

Difficulty 3/5 1-2 days Newbie friendliness 30/100

mosaicml/streaming#844 · 3 comments · Opened Dec 5, 2024
Support for Lance Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 20/100

mosaicml/streaming#832 · 5 comments · Opened Nov 20, 2024
Pipeline Parallelism (Supported? How to?) Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

mosaicml/streaming#827 · 6 comments · Opened Nov 14, 2024
Cannot Load MDS Dataset Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 35/100

mosaicml/streaming#824 · 43 comments · Opened Nov 11, 2024
UnicodeDecodeError: ... Efficient way to debug the dataset with streaming? Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 20/100

mosaicml/streaming#820 · 4 comments · Opened Nov 1, 2024
write to S3 is very slow Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mosaicml/streaming#812 · 3 comments · Opened Oct 25, 2024
Choose JPEG compression level Open

enhancement

Difficulty 2/5 1-3 hours Newbie friendliness 45/100

mosaicml/streaming#811 · 1 comment · Opened Oct 24, 2024
MosaicML-Streaming on Databricks Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

mosaicml/streaming#801 · 9 comments · Opened Oct 10, 2024
Support for on-the-fly filtering Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

mosaicml/streaming#800 · 1 comment · 8 reactions · Opened Oct 9, 2024
Make `epoch_sample_ids` cachable Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

mosaicml/streaming#792 · 2 comments · Opened Sep 28, 2024
Dataset does not work after stopping training Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 30/100

mosaicml/streaming#781 · 1 comment · Opened Sep 15, 2024
Sparse Numpy Arrays Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

mosaicml/streaming#780 · Opened Sep 10, 2024
JointWriter: Allow shard file appending Open

bug

Difficulty 5/5 Over a week Newbie friendliness 35/100

mosaicml/streaming#775 · 2 comments · Opened Sep 5, 2024
MemoryError: Unable to allocate Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 28/100

mosaicml/streaming#771 · 1 comment · Opened Sep 2, 2024
File exists: '/000000_epoch_shape' when using the ddp strategy from pytorch lightning Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 25/100

mosaicml/streaming#767 · 23 comments · 1 reaction · Opened Aug 25, 2024
Estimate total shards at the beginning of data conversion Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 38/100

mosaicml/streaming#742 · 1 comment · Opened Aug 3, 2024
huge temp files while uploading data using MDS writer Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 30/100

mosaicml/streaming#734 · 2 comments · Opened Jul 24, 2024
Download optimal for device_per_stream batching method. Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

mosaicml/streaming#726 · 4 comments · Opened Jul 16, 2024
Replication changes sample order Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 38/100

mosaicml/streaming#725 · 3 comments · Opened Jul 15, 2024
'File exists: "/00000_locals"' when integrated with deepspeed training scripts Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 28/100

mosaicml/streaming#717 · 4 comments · 3 reactions · Opened Jul 8, 2024
All processes allocate memory on rank 0 during StreamingDataset initialization in a distributed setting Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mosaicml/streaming#716 · 4 comments · Opened Jul 2, 2024

mosaicml/streaming

Get new issues in your inbox