C++: Serialization of bit arrays (both fixed and variable-length) is slow

Open
#285 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
cpp
Domain
performance

Research direction

The issue identifies C++ bit array serialization and mentions VariableLengthArray<bool> from #284, but gives no file paths or tests. Start by locating the C++ serialization methods for fixed and variable length bit arrays, then compare their behavior with the C and Python approaches described here. Done means finding an approach that improves serialization without requiring custom bit containers and preserves compatibility with standard containers.

Written by the indexing model from the issue text.

Description

under consideration

Large bit arrays can be found even in the standard data type set, so this matter may potentially affect common applications.

In C, the memory storage format of bit arrays is the same as their wire representation, which enables serialization via memcpy. The application can manipulate the contents using nunavutCopyBits, nunavutSetBit, and nunavutGetBit.

In Python, bit arrays are stored using NumPy arrays and serialized using numpy.packbits.

In C++, the current implementation (de)serializes arrays bit-by-bit which is likely to cause performance issues. Sadly the C++ implementation cannot enforce a wire-compatible memory storage format because it has to be compatible with standard containers like std::vector<bool> and std::bitset. Are there any ideas on how to improve the bit array serialization without requiring the use of custom bit containers where the memory storage format is known?

In the specialization of VariableLengthArray<bool> that I implemented in #284 the memory storage format is the same as the wire format but currently, the serialization methods cannot benefit from that.

Dominant language
Python
Stars
48
Forks
26
Avg merge
1d 18h
Merged PRs (30d)
9

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from OpenCyphal/nunavut

All issues in OpenCyphal/nunavut

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.