[Python][Docs] Opening a partitioned dataset with schema and filter

Open Beginner friendly
#30,800 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
72/100
Issue type
Documentation
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
documentation

Research direction

Start by locating the Python dataset-opening documentation covering partitioning, schemas, and filters. Add the provided partitioned-dataset example or a concise note explaining that the schema must include partitioning names when filters reference them, and verify that the documented behavior matches the example.

Written by the indexing model from the issue text.

Description

Component: Documentation Component: Python good-first-issue Status: needs champion Type: enhancement

Add a note to the docs that if partitioning and schema are both specified at opening of a dataset and partitioning names are not included in the data, schema needs to include the partitioning names (directory or hive partitioning) in a case that filtering will be done.

Example:


import numpy as np
import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as ds

# Define the data
table = pa.table({'one': [-1, np.nan, 2.5],
                   'two': ['foo', 'bar', 'baz'],
                   'three': [True, False, True]})

# Write to partitioned dataset
# The files will include columns "two" and "three"
pq.write_to_dataset(table, root_path='dataset_name',
                    partition_cols=['one'])

# Reading the partitioned dataset with schema not including partitioned names
# will error

schema = pa.schema([("three", "double")])
data = ds.dataset("dataset_name", partitioning="hive", schema=schema)
subset = ds.field("one") == 2.5
data.to_table(filter=subset)

# And will not if done like so:
schema = pa.schema([("three", "double"), ("one", "double")])
data = ds.dataset("dataset_name", partitioning="hive", schema=schema)
subset = ds.field("one") == 2.5
data.to_table(filter=subset)

Reporter: Alenka Frim / @AlenkaF

Note: This issue was originally created as ARROW-15311. Please see the migration documentation for further details.

Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
93

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/arrow

All issues in apache/arrow

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.