batch(0) silently returns one batch with the whole dataset instead of raising

Open Beginner friendly
#8,472 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
68/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
python
Domain
data

Research direction

Start at Dataset.batch and IterableDataset.batch, then inspect the existing _check_batch_size helper from #8446. Reproduce the issue with batch sizes 0, -1, and 2; done means both paths reject non-positive sizes with the specified ValueError while the positive case remains unchanged.

Written by the indexing model from the issue text.

Description

Describe the bug

Dataset.batch() and IterableDataset.batch() accept batch_size=0 and negative sizes, and quietly return one batch holding the whole dataset. Asking for a zero-sized batch produces the largest possible batch, which is as close to the opposite of the request as the API can get.

Steps or code to reproduce the bug
from datasets import Dataset

ds = Dataset.from_dict({"a": list(range(5))})
it = ds.to_iterable_dataset()

[len(b["a"]) for b in it.batch(0)]    # [5]
[len(b["a"]) for b in it.batch(-1)]   # [5]
[len(b["a"]) for b in it.batch(2)]    # [2, 2, 1]   <- sane case

len(ds.batch(0))    # 1
len(ds.batch(-1))   # 1
len(ds.batch(2))    # 3

Both the eager and the streaming path behave the same way, so this is consistent — just consistently wrong.

Expected behavior

batch_size should be required to be a positive integer, raising something like

ValueError: batch_size must be a positive integer, but got 0.

Silently substituting "everything in one batch" is the worst of the options: it looks like it worked, and a batch_size computed at runtime that comes out as 0 (an empty shard, an integer division, a config default that did not get filled in) turns into a single unbounded batch. On a large dataset that is a memory blow-up rather than an error.

Note on overlap with open PRs

I checked the open PRs before filing, and this specific path is not covered by either of the two nearby ones:

  • #8446 adds _check_batch_size but wires it only into the export methods (to_csv, to_json, ...), not batch().
  • #8443 validates by_column and the "missing batch_size or by_column" message in Dataset.batch, but does not check the value of batch_size.

So the helper that batch() wants already exists in #8446:

def _check_batch_size(batch_size: Optional[int]):
    if batch_size is not None and batch_size <= 0:
        raise ValueError(f"batch_size must be a positive integer, but got {batch_size}.")

and the fix is largely calling it from Dataset.batch and IterableDataset.batch too. That does mean this is best done after #8446 lands, to avoid two copies of the same helper.

Related, lower confidence

Same family, listed separately because the right answer is less obvious and I am not proposing a change: IterableDataset.take(-1) yields 0 rows and skip(-1) yields all rows, i.e. negatives are silently clamped rather than rejected. That is defensible as "clamp like a slice", unlike batch(0), where there is no reading under which one giant batch is the answer.

Environment info
  • datasets version: 5.0.2.dev0 (main @ 48b7ee7)
  • Python version: 3.11.9
  • Platform: Windows 11
  • PyArrow version: 25.0.1
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
4d 4h
Merged PRs (30d)
15

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from huggingface/datasets

All issues in huggingface/datasets

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.