csp::Struct leaks one StructMeta reference per construction

Read cpp/csp/engine/Struct.h around ~Struct() and HiddenData, then inspect cpp/csp/engine/Struct.cpp around construction and operator delete. The report proposes destroying HiddenData after destroy() and includes passing test-suite results; reproduce with the provided construction loop and run PYTHONMALLOC=malloc pytest csp/tests/test_engine.py csp/tests/test_baselib.py csp/tests/impl. Done means the meta use-count no longer grows after freed structs and both repro scripts finish without crashing.

Describe the bug

Every csp::Struct takes a strong reference to its StructMeta and never releases it. The reference sits in the hidden header that Struct::Struct placement-constructs, and neither ~Struct() nor Struct::operator delete destroys that header. As a result:

• The meta's std::shared_ptr use-count goes up by one for every struct ever constructed. That includes copy(), deepcopy() and unpickling.

• The count never goes down when structs are freed.

• libstdc++ stores the count as a 32-bit _Atomic_word, so after about 2³² constructions of one struct type in one process it wraps.

• At the wrap, a shared_from_this() temporary in createRaw() releases what looks like the last reference. The StructMeta is destroyed and freed, and initialize() then reads its freed field table. The result is a SIGSEGV, or silent heap corruption.

Affected: 0.17.0 through 0.19.1 (current main, b433025), verified. The same code is already in v0.12.0, so older versions are very likely affected too.

The code, at v0.19.1 (the line numbers are identical in v0.17.0):

• cpp/csp/engine/Struct.cpp:618–626, Struct::Struct: new( hidden() ) HiddenData(); … hidden() -> meta = meta; adds one reference.

• cpp/csp/engine/Struct.h:846–851: struct HiddenData { size_t refcount; std::shared_ptr<const StructMeta> meta; void * dialectPtr; };, stored just before this.

• cpp/csp/engine/Struct.h:819–822: ~Struct() { meta() -> destroy( this ); }. There is no ~HiddenData().

• cpp/csp/engine/Struct.cpp:640–644, Struct::operator delete: only ::operator delete( p ).

• cpp/csp/engine/Struct.cpp:222: s = new ( shared_from_this() ) Struct( shared_from_this() );. Suppose the count is −2 (as a 32-bit int) before this line:

  • The two temporaries take it to −1 and then 0. _M_add_ref_lock only refuses when the count is exactly 0 on entry.

  • The header copy takes it to 1.

  • The first temporary's destructor sees 1 → 0 and runs _M_dispose() + _M_destroy(), which frees the whole make_shared<DialectStructMeta> block (cpp/csp/python/PyStruct.cpp:254).

  • The second temporary's destructor then decrements freed memory.

• cpp/csp/engine/Struct.cpp:299–322: StructMeta::initialize( s ) runs on the freed meta. It loads m_fields[idx].get() from the freed vector buffer and calls through that pointer's vtable. The fault address is whatever the allocator left in that slot; with glibc safe-linking and an empty tcache bin it is buffer_address >> 12, a small unaligned value.

Passing 2³¹ is harmless, because the count simply goes negative. The crash comes at construction number 2³² − 2 − k, where k is the number of persistent holders of the meta (the PyStructMeta itself plus cached CspStructTypes, usually 1–2). It happens on whichever thread builds that struct.

To Reproduce

Shortcut (about 1 s): set the counter just below the wrap point, then construct.

import ctypes
import sys
import csp
class S(csp.Struct):

a: str
def use_count(struct_type):

ctrl = ctypes.c_uint64.from_address(id(struct_type) + type.basicsize + 8).value

return ctypes.c_int32.from_address(ctrl + 8)
print(csp.version, sys.version.split()[0], sys.platform)

uc = use_count(S)

start = uc.value

for _ in range(1000):

S(a="x")

print(f"use_count {start} -> {uc.value} after 1000 constructions, all freed")
uc.value = -7

for i in range(10):

print(f"construction {i}: use_count before = {uc.value}", flush=True)

S(a="x")

print("no crash")

The shortcut assumes this memory layout:

• PyStructMeta is a PyHeapTypeObject followed by std::shared_ptr<StructMeta> structMeta (cpp/csp/python/PyStruct.h:14–20). The shared_ptr therefore starts at offset type.basicsize (that is, sizeof(PyHeapTypeObject)), and its second word is the control-block pointer.

• The libstdccontrol block (std::_Sp_counted_base) is a vptr followed by _Atomic_word _M_use_count, a 32-bit int at offset 8. This holds for the manylinux wheels. libc builds (macOS) use a different layout and encoding, so the offsets would need changing there.

• Writing −7 is equivalent to having built 2³² − 7 structs, because the counter is a 32-bit int whose atomic increments wrap.

No-shortcut variant: no ctypes and no layout assumptions. It takes about 5 minutes on one core at roughly 14 M constructions/s, and memory stays flat because every struct is freed immediately.

import collections

import itertools

import csp
class S(csp.Struct):

a: str
for n in range(1, 17):

collections.deque(itertools.starmap(S, itertools.repeat((), 1 << 28)), maxlen=0)

print(f"{n << 28:,} constructions", flush=True)

print("no crash")

16 × 2²⁸ = 2³², so it crashes partway through the last chunk.

Expected behavior

Constructing and freeing structs should leave the meta's use-count unchanged, and a process should be able to construct any number of structs of a type. The shortcut's first line should read use_count 1 -> 1, and both scripts should end with no crash.

Error Message

The shortcut on the csp 0.19.1 wheel from PyPI: the use-count goes up by one per construction and nothing is released, then it crashes on the construction where the count was −2.

0.19.1 3.13.12 linux

use_count 1 -> 1001 after 1000 constructions, all freed

construction 0: use_count before = -7

construction 1: use_count before = -6

construction 2: use_count before = -5

construction 3: use_count before = -4

construction 4: use_count before = -3

construction 5: use_count before = -2

signal 11 (Segmentation fault), address is 0xe100000020 from

[bt]: (0) printBacktrace()

[bt]: (1) sigabrt_handler(int, siginfo_t*, void*)

[bt]: (2) /lib/x86_64-linux-gnu/libc.so.6(+0x45330) [0x7fa3c1245330]

[bt]: (3) csp::StructMeta::initialize(csp::Struct*) const

[bt]: (4) csp::StructMeta::createRaw() const

[bt]: (5) csp::python::PyStruct_new(_typeobject*, _object*, _object*)

[bt]: (6) python() [0x16034c1]

[bt]: (7) python(_PyEval_EvalFrameDefault+0xc341) [0x1621547]

[bt]: (8) python(PyEval_EvalCode+0x94) [0x16a9b14]

...

The fault address changes from run to run (0, 0xe100000020, 0x1255f, 0x1ce8d, …) depending on what the allocator wrote into the freed buffer. The no-shortcut variant on 0.19.1 crashed after 4,026,531,840 + some fraction of 2²⁸ constructions (about 300 s), with the same frames (3)–(6).

The same shortcut against an ASan build of v0.19.1 (CSP_ENABLE_ASAN=1, Debug), trimmed:

ERROR: AddressSanitizer: heap-use-after-free on address 0x51200064a2c8

READ of size 8 at 0x51200064a2c8 thread T0

#0 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()   bits/shared_ptr_base.h:337

#3 std::shared_ptr<csp::StructMeta const>::~shared_ptr()              bits/shared_ptr.h:175

#4 csp::StructMeta::createRaw() const                                 cpp/csp/engine/Struct.cpp:222

#5 csp::StructMeta::create() const                                    cpp/csp/engine/Struct.h:665

#6 csp::python::PyStruct_new(_typeobject*, _object*, _object*)        cpp/csp/python/PyStruct.cpp:791

0x51200064a2c8 is located 8 bytes inside of 312-byte region [0x51200064a2c0,0x51200064a3f8)

freed by thread T0 here:

#0 operator delete(void*, unsigned long)

#3 std::_Sp_counted_ptr_inplace<csp::python::DialectStructMeta, ...>::_M_destroy()   bits/shared_ptr_base.h:623

#4 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use()         bits/shared_ptr_base.h:191

#6 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()                  bits/shared_ptr_base.h:353

#9 std::shared_ptr<csp::StructMeta const>::~shared_ptr()                             bits/shared_ptr.h:175

#10 csp::StructMeta::createRaw() const                                               cpp/csp/engine/Struct.cpp:222

previously allocated by thread T0 here:

#0 operator new(unsigned long)

#6 std::make_shared<csp::python::DialectStructMeta, ...>

#7 PyStructMeta_new                                                                  cpp/csp/python/PyStruct.cpp:254

Runtime Environment

0.19.1                (also reproduced on 0.17.0)

3.13.12 (main, Mar  3 2026, 15:01:51) [Clang 21.1.4]   python-build-standalone, x86_64

linux                 (Ubuntu 24.04 / glibc 2.39 for the repros; also seen on Debian 13 / glibc 2.41)

Additional context

Impact. Any long-running process crashes after about 4.29×10⁹ constructions of a single struct type, no matter which thread does the constructing, how fast, or which allocator is in use. Restarting the process resets the count.

We run a realtime engine that builds one small struct type (two str fields) 100–200k times per second at peak. That is about once a day at our rate. It crashed repeatedly in StructMeta::initialize ← createRaw ← PyStruct_new, and three independent crashes happened at the same cumulative construction count to within 0.2%. Stress tests that restarted more often, or spread constructions over many types, never reproduced it.

Proposed fix (against v0.19.1; it applies unchanged to v0.17.0):

--- a/cpp/csp/engine/Struct.h

+++ b/cpp/csp/engine/Struct.h

@@ -819,6 +819,7 @@ private:

~Struct()

{

meta() -> destroy( this );



   hidden() -> ~HiddenData();

}
void incref() { ++hidden() -> refcount; }

HiddenData has to be destroyed after destroy(), which still needs meta(), and before operator delete frees the block. If this releases the last reference, for example for a C++-only or otherwise dropped struct type, the meta is destroyed only after all field destructors have run. Python-defined types are still kept alive by the existing PyStructMeta ↔ DialectStructMeta strong cycle (cpp/csp/python/PyStruct.cpp:232–248), so their lifetime doesn't change; the counter just stops drifting.

Test evidence. The builds used gcc 13, DEBUG=1, CSP_ENABLE_ASAN=1, with the arrow/kafka/parquet/websocket adapters not built. Tests were run with PYTHONMALLOC=malloc pytest csp/tests/test_engine.py csp/tests/test_baselib.py csp/tests/impl:

Source | Without patch | With patch
-- | -- | --
v0.17.0 | 831 passed, 38 skipped, 3 xfailed | 831 passed, 38 skipped, 3 xfailed
v0.19.1 | 843 passed, 38 skipped, 3 xfailed, 5 subtests passed | 843 passed, 38 skipped, 3 xfailed, 5 subtests passed

None of these runs produced an ASan report. With the patch:

• The shortcut prints use_count 1 -> 1 and no crash.

• Holding 1,000 structs raises the count by 1,000, and freeing them brings it back.

• copy() leaves the count unchanged overall.

Regression test. There is no C++ struct test under cpp/tests/engine/ yet. A small gtest could build a StructMeta with one StringStructField, call meta->create() N times, and assert that meta.use_count() is unchanged once those structs are gone. Alternatively, expose the count to Python (for example a use_count entry in StructMeta._metadata_info()) and assert in csp/tests/impl/test_struct.py that it stays flat across N constructions, copy(), deepcopy() and pickle round-trips.