Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

IPC `ArrowFileWriter` can produce entirely zero-filled encapsulated messages

Aperta
  • Qualcosa non va?
#409 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
38/100
Tipo di issue
Bug
Chiarezza
Da chiarire
Stato di attività
Tranquilla
Stack tecnologico
csharp
Ambito
data

Direzione di ricerca

Inizia leggendo ArrowFileWriter e i punti di utilizzo in ArrowWriter.cs e ArrowBatchWriter.cs, in particolare BaseStream.Position e le chiamate serializzate a WriteRecordBatch da thread diversi. Se possibile, riproduci il messaggio IPC riempito di zeri, confrontando le scritture compresse e non compresse e gli offset interessati. Il lavoro è completato quando viene identificato se ArrowFileWriter può far avanzare la posizione senza scrivere, oppure vengono documentati l’uso richiesto dello stream o la causa ambientale.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Describe the bug, including details regarding any error messages, version, and platform.

Environment

  • Apache.Arrow / Apache.Arrow.Compression 23.0.0, net472, x64, Windows 11
  • ArrowFileWriter over a plain FileStream, writing record batches continuously for the duration of a recording

Background

We are using Apache.Arrow to save continuous streaming data from ONIX hardware, where our project (OpenEphys.Onix1) is hosted as a package that can run in Bonsai-Rx.

Problem

Rarely, on one specific computer, a file comes out with an entire encapsulated message (continuation marker, metadata, and body) replaced by zeros. The footer is intact and its Block offsets are correct for every batch, including all batches after the zeroed one, so the file reads until it hits the gap and then raises ArrowInvalid: Unexpected empty message in IPC file format. Happens with and without Zstd compression, always early in the recording. Individual valid batches can be read and concatenated together, but this leaves a gap in the recorded samples as confirmed by our clock parameter.

ArrowFileWriter takes the block offset from BaseStream.Position and the following batch is recorded at the correct offset; we believe this points to Arrow having issued the writes and FileStream having accepted them, but the bytes were lost before being saved to disk. We'd like to confirm if this is the case before concluding it's environmental.

We are tracking this in our own issue in case it is how we are utilizing the API. For more details (e.g., byte offsets, and affected files), see open-ephys/bonsai-onix1#683. We have been unable to replicate this on two other systems.

Our usage

  • ArrowWriter.cs — construction, WriteRecordBatch, and WriteEnd on dispose
  • ArrowBatchWriter.cs — batches are accumulated with an Rx Buffer(timeout, count), so WriteRecordBatch calls are serialized but arrive on different threads (producer thread on a count flush, thread-pool timer thread on a timeout flush)

Note also that our ArrowBuffers wrap unmanaged memory via a custom MemoryManager<byte> to handle OpenCV.Net.Mat objects.

I'd be happy to answer any questions you might have about our usage, and if there is any other information I can provide to narrow this down please let me know.

Questions

  1. Is there any path in ArrowFileWriter that advances BaseStream.Position without writing bytes (a seek, or padding emitted by seeking)?
  2. Is there anything we should be doing differently? FileOptions.WriteThrough, periodic Flush(true) on the base stream, or otherwise to keep a lower layer from silently dropping a write like this?
Lingua principale
C#
Stelle
41
Fork
31
Merge medio
8h 2m
PR unite (30g)
15

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/arrow-dotnet

Tutte le issue di apache/arrow-dotnet

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.