WindowShuffleMapDataset duplicates and omits records in incomplete windows
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 35/100
Research direction
Start with WindowShuffleMapDataset._shuffled_index and compare its window calculations with WindowShuffleIterDataset(..., reverse=False). Reproduce the five-record example, then check scalar and batched access across repeated epochs; done means each epoch preserves every record, full-window seeded order is unchanged, and the first epoch agrees with the iterator variant. A linked pull request is already open.
Written by the indexing model from the issue text.
Description
WindowShuffleMapDataset can duplicate and omit records when the parent length is not divisible by window_size. This changes the training data within an epoch without raising an error.
Reproduced on d2adadec1b275251b098215abe5c6df8b16ed6da (Python 3.12, macOS arm64):
import grain
parent = grain.MapDataset.range(5)
shuffled = grain.experimental.WindowShuffleMapDataset(
parent, window_size=4, seed=42
)
print(list(shuffled))
# [0, 1, 2, 3, 1]
Expected: a permutation of [0, 1, 2, 3, 4], with records staying in their windows. Record 4 is missing and record 1 appears twice. The corresponding WindowShuffleIterDataset(..., reverse=False) preserves all five records.
_shuffled_index uses window_size - 1 as the shuffle bound even for an incomplete final window. An index beyond the parent length then wraps into the next epoch. The global window boundaries also drift across repeated epochs. With 1,003 records and a window of 1,000, seed 42 omits records 1,000–1,002 in the first epoch; the second epoch has six missing records. A 12-record, four-record-window control preserves every record in three consecutive epochs.
The map variant needs to bound the last window to the remaining records and compute window boundaries within each parent epoch. Full-window datasets should retain their existing seeded order. Regression coverage should include scalar and batched access, repeat, and agreement with the iterator variant for the first epoch.
- Dominant language
- Python
- Stars
- 795
- Forks
- 91
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 15
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from google/grain
-
ConcatThenSplitIterDataset drops a document when restored from a JSON-serialized statePossibly taken @AshishKumar4 claimed this 4 days ago. Opentype:bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
Closing an mp_prefetch iterator stops its worker processes one at a timePossibly taken @jabrailkhalil claimed this 11 days ago. Opentype:bug
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
Maintainers usually reply within 1 day
-
`stats._iter_weakref_registry` keeps refs of dead iterators, and a new iterator at a reused address is not registeredPossibly taken @Shr3yash claimed this 11 days ago. Opentype:bug
Difficulty 3/5 Half a day Newbie friendliness 55/100
Maintainers usually reply within 1 day
-
type:bug
Difficulty 4/5 3-5 days Newbie friendliness 25/100
Maintainers usually reply within 1 day
-
mp_prefetch: a reference cycle per finished shard, and SharedMemory closed by two threadsPossibly taken @mahdi-shafiei claimed this 15 days ago. Opentype:bug
Difficulty 4/5 3-5 days Newbie friendliness 20/100
Maintainers usually reply within 1 day
Similar issues
-
good first issue infrastructure
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
kuznia-rdzeni/coreblocks#1135 ·
-
good first issue server
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
documentation priority: high
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
a-lex-s/mealie-telegram-bot#54 ·
Maintainers usually reply within 1 day
-
type/test
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
SaltedTan/virgil-language-server#218 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
btclib-org/btclib-wallet#541 ·
Maintainers usually reply within 1 day