[R] Silent loss of data integrity for date-time and numeric attributes in write_parquet()/read_parquet()
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 72/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- r
- Domain
- data-engineering
Research direction
Start at the R metadata serialization used by write_parquet() and read_parquet(), focusing on the serialize() call in the suggested fix. Confirm that double-valued attributes, including date-times, round-trip exactly through write_parquet()/read_parquet(), while existing files remain readable and no reader change is needed.
Written by the indexing model from the issue text.
Description
Describe the bug, including details regarding any error messages, version, and platform.
Double-valued attributes, including date-times, do not round-trip through Arrow's R metadata. No warning is raised. (Tested on arrow 25.0.1, R 4.6.1, Linux.)
library(arrow)
options(digits.secs = 6)
df <- data.frame(x = 1:3)
attr(df, "updated") <- as.POSIXct("2026-10-02 14:30:15.987654321", tz = "UTC")
f <- tempfile(fileext = ".parquet")
write_parquet(df, f)
back <- read_parquet(f)
attr(df, "updated")
#> [1] "2026-10-02 14:30:15.987654 UTC"
attr(back, "updated")
#> [1] "2026-10-02 14:30:15.987653 UTC"
identical(attr(back, "updated"), attr(df, "updated"))
#> [1] FALSE
Suggested fix
Fortuitously, R already has an exact variant of the ASCII format, so the fix is simple and fully backwards compatible.
Arrow uses R_pstream_ascii_format (ascii = TRUE in base::serialize). Instead use R_pstream_asciihex_format (ascii = NA).
out <- serialize(safe_r_metadata(x, on_save = TRUE), NULL, ascii = NA)
The hex format means that doubles are written out in hex instead of decimal.
Either way, the output still starts with "A" (arrow's check) and base unserialize() reads it fine, so the reader needs no change and previous versions work with the output.
Component(s)
R
- Dominant language
- C++
- Stars
- 17.2k
- Forks
- 4.3k
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 93
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/arrow
-
[C++][Parquet] Plaintext-footer files written with AES_GCM_CTR_V1 record AES_GCM_V1 as the encryption algorithm and cannot be readPossibly taken @YusefSyed claimed this 3 days ago. OpenType: bug
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
[R] Expose ignore_extra_columns and pad_short_rows CSV parse optionsPossibly taken A pull request linked to this issue is open or already merged. OpenComponent: R good-first-issue Type: enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
Component: C++
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
Maintainers usually reply within 1 day
-
Component: C++ Type: enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Maintainers usually reply within 1 day
-
Component: GLib
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Maintainers usually reply within 1 day
Similar issues
-
code-quality libc++
Difficulty 1/5 Under an hour Newbie friendliness 82/100
llvm/llvm-project#229284 ·
Maintainers usually reply within 1 day
-
test-issue
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
llvm/offload-test-suite#1557 ·
Maintainers usually reply within 1 day
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
iOS: hidden scale bar invalidates its intrinsic content size on every layout pass of MLNMapViewOpen
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
maplibre/maplibre-native#4723 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
HarbourMasters/Shipwright#7320 ·
Maintainers usually reply within 1 day