IPC `ArrowFileWriter` can produce entirely zero-filled encapsulated messages
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 38/100
Research direction
Start by reading ArrowFileWriter and the usage points in ArrowWriter.cs and ArrowBatchWriter.cs, especially BaseStream.Position and the serialized WriteRecordBatch calls from different threads. Reproduce the zero-filled IPC message if possible, comparing compressed and uncompressed writes and the affected offsets. Done means identifying whether ArrowFileWriter can advance the position without writing, or documenting the required stream usage or environmental cause.
Written by the indexing model from the issue text.
Description
Describe the bug, including details regarding any error messages, version, and platform.
Environment
Apache.Arrow/Apache.Arrow.Compression23.0.0,net472, x64, Windows 11ArrowFileWriterover a plainFileStream, writing record batches continuously for the duration of a recording
Background
We are using Apache.Arrow to save continuous streaming data from ONIX hardware, where our project (OpenEphys.Onix1) is hosted as a package that can run in Bonsai-Rx.
Problem
Rarely, on one specific computer, a file comes out with an entire encapsulated message (continuation marker, metadata, and body) replaced by zeros. The footer is intact and its Block offsets are correct for every batch, including all batches after the zeroed one, so the file reads until it hits the gap and then raises ArrowInvalid: Unexpected empty message in IPC file format. Happens with and without Zstd compression, always early in the recording. Individual valid batches can be read and concatenated together, but this leaves a gap in the recorded samples as confirmed by our clock parameter.
ArrowFileWriter takes the block offset from BaseStream.Position and the following batch is recorded at the correct offset; we believe this points to Arrow having issued the writes and FileStream having accepted them, but the bytes were lost before being saved to disk. We'd like to confirm if this is the case before concluding it's environmental.
We are tracking this in our own issue in case it is how we are utilizing the API. For more details (e.g., byte offsets, and affected files), see open-ephys/bonsai-onix1#683. We have been unable to replicate this on two other systems.
Our usage
ArrowWriter.cs— construction,WriteRecordBatch, andWriteEndon disposeArrowBatchWriter.cs— batches are accumulated with an RxBuffer(timeout, count), soWriteRecordBatchcalls are serialized but arrive on different threads (producer thread on a count flush, thread-pool timer thread on a timeout flush)
Note also that our ArrowBuffers wrap unmanaged memory via a custom MemoryManager<byte> to handle OpenCV.Net.Mat objects.
I'd be happy to answer any questions you might have about our usage, and if there is any other information I can provide to narrow this down please let me know.
Questions
- Is there any path in
ArrowFileWriterthat advancesBaseStream.Positionwithout writing bytes (a seek, or padding emitted by seeking)? - Is there anything we should be doing differently?
FileOptions.WriteThrough, periodicFlush(true)on the base stream, or otherwise to keep a lower layer from silently dropping a write like this?
- Dominant language
- C#
- Stars
- 41
- Forks
- 31
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 15
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/arrow-dotnet
-
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
apache/arrow-dotnet#410 · 4 comments ·
Maintainers usually reply within 1 day
-
ArrowArrayConcatenator: null arrays are refused, and dictionary arrays lose their dictionaryPossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 4/5 3-5 days Newbie friendliness 68/100
apache/arrow-dotnet#446 ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
apache/arrow-dotnet#397 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 55/100
apache/arrow-dotnet#378 ·
Maintainers usually reply within 1 day
-
Add compute kernels (Sum/Min/Max/Mean) using System.Numerics.Tensors (TensorPrimitives) and Tensor<T> supportPossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 5/5 Over a week Newbie friendliness 32/100
apache/arrow-dotnet#375 · 1 comment ·
Maintainers usually reply within 1 day
All issues in apache/arrow-dotnet
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
PCL-Community/PCL-CE#3652 ·
Maintainers usually reply within 1 day