Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

ConcatThenSplitIterDataset drops a document when restored from a JSON-serialized state

Open Beginner friendly
  • Is something wrong?
#1,433 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@AshishKumar4 is already working on this.

Since Oct 19, 2026.

  • #1434 by @AshishKumar4 — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
75/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
data-engineering

Research direction

Start by locating _CtsState, _EMPTY_SLICE, and the slice comparisons in ConcatThenSplitIterDataset; the issue identifies _CtsState.__post_init__ as one possible fix. Run the JSON serialization reproduction, then check that restoring its state produces the same packed elements as an uninterrupted run.

Written by the indexing model from the issue text.

Description

_CtsState.remainder_slices marks "no remainder" (or a whole, unsliced remainder) with the tuple
sentinel _EMPTY_SLICE = (-1, -1), and has_remainder, _CtsElement.split and
get_sliced_features compare slices against it with !=. A state that goes through JSON, as
checkpoints do, comes back with lists ([-1, -1]), which never equal the tuple. set_state then
takes the next parent element as a remainder and get_sliced_features slices it [-1:-1], so that
element is dropped from the packed stream.

Reproduction (grain 0.2.18; the comparisons are the same on main):

import json
import numpy as np
import grain.python as pygrain
import grain.experimental as gx

docs = [{"text": (np.arange(n) + 100 * i).astype(np.int32)} for i, n in enumerate((3, 5, 8, 4, 6))]

def build():
    return iter(gx.ConcatThenSplitIterDataset(pygrain.MapDataset.source(docs).to_iter_dataset(),
                                              length_struct={"text": 8}))

whole = build()
windows = [next(whole)["text"].tolist() for _ in range(4)]
stopped = build()
next(stopped)
state = json.loads(json.dumps(stopped.get_state()))  # {'remainder_slices': {'text': [-1, -1]}, ...}
resumed = build()
resumed.set_state(state)
print([next(resumed)["text"].tolist() for _ in range(3)], windows[1:])  # StopIteration: a document was dropped

With state["remainder_slices"] converted back to tuples the restore matches the uninterrupted run.
A fix: normalize the slices to tuples in _CtsState.__post_init__ (or compare tuple(sl)).

Dominant language
Python
Stars
792
Forks
89
Avg merge
1d 10h
Merged PRs (30d)
16

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from google/grain

All issues in google/grain

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.