C++: Serialization of bit arrays (both fixed and variable-length) is slow
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- cpp
- Domain
- performance
Research direction
The issue identifies C++ bit array serialization and mentions VariableLengthArray<bool> from #284, but gives no file paths or tests. Start by locating the C++ serialization methods for fixed and variable length bit arrays, then compare their behavior with the C and Python approaches described here. Done means finding an approach that improves serialization without requiring custom bit containers and preserves compatibility with standard containers.
Written by the indexing model from the issue text.
Description
Large bit arrays can be found even in the standard data type set, so this matter may potentially affect common applications.
In C, the memory storage format of bit arrays is the same as their wire representation, which enables serialization via memcpy. The application can manipulate the contents using nunavutCopyBits, nunavutSetBit, and nunavutGetBit.
In Python, bit arrays are stored using NumPy arrays and serialized using numpy.packbits.
In C++, the current implementation (de)serializes arrays bit-by-bit which is likely to cause performance issues. Sadly the C++ implementation cannot enforce a wire-compatible memory storage format because it has to be compatible with standard containers like std::vector<bool> and std::bitset. Are there any ideas on how to improve the bit array serialization without requiring the use of custom bit containers where the memory storage format is known?
In the specialization of VariableLengthArray<bool> that I implemented in #284 the memory storage format is the same as the wire format but currently, the serialization methods cannot benefit from that.
- Dominant language
- Python
- Stars
- 48
- Forks
- 26
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 9
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from OpenCyphal/nunavut
-
build and test dependencies
Difficulty 4/5 3-5 days Newbie friendliness 55/100
OpenCyphal/nunavut#419 ·
Maintainers usually reply within 1 day
-
build and test C++ dependencies
Difficulty 4/5 3-5 days Newbie friendliness 50/100
OpenCyphal/nunavut#418 ·
Maintainers usually reply within 1 day
-
build and test Python
Difficulty 4/5 3-5 days Newbie friendliness 45/100
OpenCyphal/nunavut#416 ·
Maintainers usually reply within 1 day
-
C C++ Python
Difficulty 4/5 3-5 days Newbie friendliness 52/100
OpenCyphal/nunavut#414 · 1 comment ·
Maintainers usually reply within 1 day
-
feature under consideration
Difficulty 5/5 Over a week Newbie friendliness 32/100
OpenCyphal/nunavut#413 ·
Maintainers usually reply within 1 day
All issues in OpenCyphal/nunavut
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 86/100
py-econometrics/pyfixest#2072 ·
Maintainers usually reply within 1 day
-
~planning docs
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
Kromatic-Innovation/athenaeum#3084 ·
Maintainers usually reply within 1 day
-
Bosch Room Thermostat II number entities offer -500 to 3000 °C setpoint limits and a -32768 to 32767 °C outdoor temperaturePossibly taken @zigpy-review-bot claimed this today. Openbot: code audit
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
zigpy/zha-device-handlers#5769 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 Half a day Newbie friendliness 80/100
R0Y00/Xbox2Windows#254 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Kludex/python-multipart#375 ·
Maintainers usually reply within 1 day