Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

readPoint() reads only part.0.parquet, so points written from a partitioned dask frame lose rows

Open Beginner friendly
  • Is something wrong?
#287 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
68/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, r
Domain
data

Research direction

Start in R/read.R at the readPoint() path that opens points.parquet/part.0.parquet; compare the single-file pandas case with Python's multiple parquet parts from a partitioned dask frame. Read all parts and check how source_path is set, since path() in R/path.R returns it. Done when the reproduced four-part example returns all 5,000,000 points and path() reports a useful source path.

Written by the indexing model from the issue text.

Description

While testing spicyR's SpatialData writer against stores written by Python, we found that readPoint() can silently return only part of a points element. It happens whenever Python wrote the points from a dask frame with more than one partition, which Python stores as several parquet files. readPoint() reads only the first. The fix looks like a one-line change to read all the parts, and we've checked that it returns every point in the example below.

What happens. readPoint() opens points.parquet/part.0.parquet only (R/read.R#L80). Python's spatialdata writes one parquet file per dask partition. A points element built from a dask frame with several partitions is therefore stored as part.0.parquet … part.k.parquet, and readPoint() returns the rows of the first part only, with no warning. (Points built from a pandas frame are a single partition and aren't affected, however large they are.)

To reproduce (spatialdata 0.7.3, spatialdataR devel at db1f01a):

import numpy as np, pandas as pd, dask.dataframe as dd, spatialdata as sd
from spatialdata.models import PointsModel
rng = np.random.default_rng(1)
df = pd.DataFrame({"x": rng.random(5_000_000), "y": rng.random(5_000_000)})
pts = PointsModel.parse(dd.from_pandas(df, npartitions=4))
sd.SpatialData(points={"pts": pts}).write("parts.zarr")
# parts.zarr/points/pts/points.parquet/ holds part.0.parquet ... part.3.parquet

spatialdataR::readPoint("parts.zarr/points/pts")
#> class: SpatialDataPoint
#> count: 1250000

The element has 5,000,000 points.

Suggested fix. Read every part. ddbs_open_dataset() accepts a glob, and with this change the example above returns all 5,000,000 points:

pq <- file.path(x, "points.parquet", "*.parquet")

One knock-on effect: path() returns attr(df, "source_path") (R/path.R#L41), so it would then return the glob. Setting source_path to the points.parquet folder instead might be clearer for users.

This is the bug we mentioned in #115. Let us know if you'd like the PR, or whether you'd rather fix it alongside other work on the readers.

Dominant language
R
Stars
37
Forks
10
Avg merge
22h 46m
Merged PRs (30d)
20

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from HelenaLC/spatialdataR

All issues in HelenaLC/spatialdataR

Similar issues

More R issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.