BUG: NeighborList export to HDF5 holds about two full copies of the data in memory
维护者通常 6 天内回复
还没有人认领这个 Issue。
评估
调研方向
Start in src/simplnx/DataStructure/IO/HDF5/NeighborListIO.hpp, especially NeighborListIO::writeData, and inspect the NeighborList store API for reading one list without copying. Compare its current output with the existing NeighborList HDF5 round-trip tests in test/H5Test.cpp; add the bounded-memory check there, guarded or skipped on Windows as described. Done means the export preserves the file layout, attributes, compression and linked counts dataset while staying under the stated extra heap limit.
由索引模型根据 Issue 内容生成。
描述
Problem
NeighborListIO::writeData copies the whole NeighborList two times before it writes it. Both copies are alive at the same time. The peak extra heap use is about two times the size of the list data.
Expected: the export uses a bounded staging buffer, plus one count per tuple. The extra memory does not depend on the size of the list data.
Actual: for a NeighborList<int32> with 4096 lists of 512 values (8388608 bytes of list data), the peak extra heap use during DataStructureWriter::WriteFile is 16901136 bytes. The test limit is 2097152 bytes.
For a large NeighborList, the copies can use more memory than the machine has. The export then fails with an out-of-memory error, or the system starts to swap.
Found at commit 01809c236527ba812055cd6ab46e142317670c77.
Steps to Reproduce
- Build
simplnx_testat commit01809c236527ba812055cd6ab46e142317670c77. - Create a
DataStructurewith aNeighborList<int32>of 4096 lists. Each list has 512int32values. - Open a new HDF5 file with
HDF5::FileIO::WriteFile. - Measure the peak heap use while
HDF5::DataStructureWriter::WriteFile(dataStructure, fileWriter)runs. - The peak is about two times the 8 MiB payload.
Cause
NeighborListIO::writeData first calls neighborList.getVectors(). This function returns a std::vector<std::vector<T>> by value. It copies every list from the store (https://github.com/BlueQuartzSoftware/simplnx/blob/01809c236527ba812055cd6ab46e142317670c77/src/simplnx/DataStructure/NeighborList.cpp#L471-L481). Then the function creates a second DataStore<T> named flattenedData of the total size, and copies all values into it. The neighborData copy is alive until the function ends. Both copies exist when WriteDataStore runs.
// Store list lengths separately so the reader can rebuild ragged vectors.
const auto neighborData = neighborList.getVectors(); // copy 1: every list
...
DataStore<T> flattenedData(totalItems, static_cast<T>(0)); // copy 2: all values, flattened
...
result = DataStoreIO::WriteDataStore<T>(datasetWriter, flattenedData);
Line 254 holds the first copy. Line 273 holds the second copy.
Verification Unit Test
File: test/H5Test.cpp (append at the end of the file).
The test replaces the global operator new and operator delete in the test executable. It counts the live heap bytes while tracking is on. First, it checks that the tracker sees a known 8 MiB allocation. Then it measures the peak growth during one WriteFile call.
Platform limit: the replaced operators only see allocations that are made by code that links to the same C++ runtime allocation functions as the test executable. This works on macOS and Linux, where libsimplnx uses the allocation functions of the executable. On Windows, the DLL boundary prevents this: libsimplnx.dll can use its own heap. The first check in the test (the probe) fails there, which marks the test as not valid. Guard or skip the test on Windows.
// -----------------------------------------------------------------------------
// Heap tracking for the NeighborList export memory test. The replaceable global
// allocation functions count live bytes only while tracking is enabled, so the
// test can measure the peak heap growth of one write call.
// -----------------------------------------------------------------------------
#if defined(__APPLE__)
#include <malloc/malloc.h>
#define SIMPLNX_TEST_USABLE_SIZE(ptr) malloc_size(ptr)
#elif defined(_WIN32)
#include <malloc.h>
#define SIMPLNX_TEST_USABLE_SIZE(ptr) _msize(ptr)
#else
#include <malloc.h>
#define SIMPLNX_TEST_USABLE_SIZE(ptr) malloc_usable_size(ptr)
#endif
#include <atomic>
#include <cstdlib>
#include <new>
namespace HeapTracker
{
std::atomic<bool> s_Enabled{false};
std::atomic<int64> s_LiveBytes{0};
std::atomic<int64> s_PeakBytes{0};
void Start()
{
s_LiveBytes = 0;
s_PeakBytes = 0;
s_Enabled = true;
}
int64 Stop()
{
s_Enabled = false;
return s_PeakBytes.load();
}
void OnAllocate(void* ptr)
{
if(ptr == nullptr || !s_Enabled.load(std::memory_order_relaxed))
{
return;
}
const int64 live = s_LiveBytes.fetch_add(static_cast<int64>(SIMPLNX_TEST_USABLE_SIZE(ptr))) + static_cast<int64>(SIMPLNX_TEST_USABLE_SIZE(ptr));
int64 peak = s_PeakBytes.load();
while(live > peak && !s_PeakBytes.compare_exchange_weak(peak, live))
{
}
}
void OnFree(void* ptr)
{
if(ptr == nullptr || !s_Enabled.load(std::memory_order_relaxed))
{
return;
}
s_LiveBytes.fetch_sub(static_cast<int64>(SIMPLNX_TEST_USABLE_SIZE(ptr)));
}
void* Allocate(std::size_t size)
{
void* ptr = std::malloc(size == 0 ? 1 : size);
if(ptr == nullptr)
{
throw std::bad_alloc();
}
OnAllocate(ptr);
return ptr;
}
void Free(void* ptr) noexcept
{
OnFree(ptr);
std::free(ptr);
}
} // namespace HeapTracker
void* operator new(std::size_t size)
{
return HeapTracker::Allocate(size);
}
void* operator new[](std::size_t size)
{
return HeapTracker::Allocate(size);
}
void* operator new(std::size_t size, const std::nothrow_t&) noexcept
{
try
{
return HeapTracker::Allocate(size);
} catch(...)
{
return nullptr;
}
}
void* operator new[](std::size_t size, const std::nothrow_t&) noexcept
{
try
{
return HeapTracker::Allocate(size);
} catch(...)
{
return nullptr;
}
}
void operator delete(void* ptr) noexcept
{
HeapTracker::Free(ptr);
}
void operator delete[](void* ptr) noexcept
{
HeapTracker::Free(ptr);
}
void operator delete(void* ptr, std::size_t) noexcept
{
HeapTracker::Free(ptr);
}
void operator delete[](void* ptr, std::size_t) noexcept
{
HeapTracker::Free(ptr);
}
void operator delete(void* ptr, const std::nothrow_t&) noexcept
{
HeapTracker::Free(ptr);
}
void operator delete[](void* ptr, const std::nothrow_t&) noexcept
{
HeapTracker::Free(ptr);
}
TEST_CASE("NeighborList IO: export uses bounded memory", "[NeighborList][HDF5]")
{
// 4096 lists x 512 int32 values = 8 MiB of list payload held by the store.
constexpr usize k_NumLists = 4096;
constexpr usize k_ValuesPerList = 512;
constexpr int64 k_PayloadBytes = static_cast<int64>(k_NumLists * k_ValuesPerList * sizeof(int32));
// A streamed export needs a bounded staging buffer plus one count per tuple, not
// copies of the whole payload. Allow a quarter of the payload as headroom.
constexpr int64 k_MaxExtraHeapBytes = k_PayloadBytes / 4;
// The tracker must observe allocations made inside libsimplnx for this test to mean anything.
HeapTracker::Start();
{
std::vector<int32> probe(static_cast<usize>(k_PayloadBytes / sizeof(int32)));
probe[0] = 1;
}
REQUIRE(HeapTracker::Stop() >= k_PayloadBytes);
DataStructure dataStructure;
auto* neighborList = NeighborList<int32>::Create(dataStructure, "NeighborList", ShapeType{k_NumLists});
REQUIRE(neighborList != nullptr);
for(usize i = 0; i < k_NumLists; i++)
{
std::vector<int32> values(k_ValuesPerList);
std::iota(values.begin(), values.end(), static_cast<int32>(i));
neighborList->setList(static_cast<int32>(i), values);
}
const fs::path filePath = GetDataDir() / "NeighborListExportMemory.dream3d";
auto fileWriter = HDF5::FileIO::WriteFile(filePath);
REQUIRE(fileWriter.isValid());
HeapTracker::Start();
Result<> writeResult = HDF5::DataStructureWriter::WriteFile(dataStructure, fileWriter);
const int64 peakExtraHeapBytes = HeapTracker::Stop();
SIMPLNX_RESULT_REQUIRE_VALID(writeResult);
INFO(fmt::format("List payload: {} bytes. Peak extra heap during export: {} bytes. Allowed: {} bytes.", k_PayloadBytes, peakExtraHeapBytes, k_MaxExtraHeapBytes));
REQUIRE(peakExtraHeapBytes <= k_MaxExtraHeapBytes);
}
Build and run only this test. Use the build directory of a configured simplnx build tree.
cmake --build <build-dir> -j --target simplnx_test
cd <build-dir> && ctest -R "^NeighborList IO: export uses bounded memory$" --output-on-failure
Output at commit 01809c236527ba812055cd6ab46e142317670c77:
test/H5Test.cpp:1675: FAILED:
REQUIRE( peakExtraHeapBytes <= k_MaxExtraHeapBytes )
with expansion:
16901136 (0x101e410) <= 2097152 (0x200000)
with message:
List payload: 8388608 bytes. Peak extra heap during export: 16901136 bytes.
Allowed: 2097152 bytes.
test cases: 1 | 1 failed
assertions: 5 | 4 passed | 1 failed
Proposed Fix
Write the lists in bounded batches. Do not call getVectors().
- In
NeighborListIO::writeData, make the companion counts array with one pass over the store. Use the per-tuple list size from the store and not a copy of each list. - Create the packed dataset with the known total item count. Write it hyperslab by hyperslab. Fill a staging buffer of a fixed size (for example 1 MiB of values) from consecutive lists, write it, and reuse it.
- Keep the compression setting that the function applies now (
datasetWriter.setCompressionLevel(...)). Chunking must stay valid for a one-dimensional dataset. - Check whether the store offers a read API for one list at a time that does not copy. If it does not, add one to the list store.
NeighborList::getVectors()stays for other callers. Check them for the same problem.
The streamed write must create the same file layout, attributes and Linked NumNeighbors Dataset link as now. The existing NeighborList round-trip tests cover this.
Notes
- The limit in the test is one quarter of the payload. This is only a headroom value. A correct streamed write needs far less than that.
- The test counts only heap use that goes through
operator new. It does not count memory that a library allocates withmallocdirectly. The measured peak (16901136 bytes) is about two times the payload (8388608 bytes). This is consistent with the two copies in the Cause section. - Only macOS was run. The Linux behavior of
malloc_usable_sizeand the Windows limit are described above and were not run.
- 主要语言
- C++
- 星标
- 18
- 派生
- 15
- 平均合并
- 4 天 23 小时
- 30 天内合并 PR
- 42
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
BlueQuartzSoftware/simplnx 的其他 Issue
-
bug
难度 2/5 1-3 小时 新手友好度 78/100
BlueQuartzSoftware/simplnx#1987 ·
维护者通常 6 天内回复
-
documentation
难度 2/5 1-3 小时 新手友好度 74/100
BlueQuartzSoftware/simplnx#1980 ·
维护者通常 6 天内回复
-
Testing
难度 2/5 1-3 小时 新手友好度 62/100
BlueQuartzSoftware/simplnx#1979 ·
维护者通常 6 天内回复
-
documentation
难度 2/5 1-3 小时 新手友好度 65/100
BlueQuartzSoftware/simplnx#1978 ·
维护者通常 6 天内回复
-
难度 2/5 1-3 小时 新手友好度 82/100
BlueQuartzSoftware/simplnx#1945 ·
维护者通常 6 天内回复
查看 BlueQuartzSoftware/simplnx 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 68/100
HarperFast/rocksdb-js#1118 ·
维护者通常 1 天内回复
-
bug triage
难度 2/5 1-3 小时 新手友好度 75/100
维护者通常 1 天内回复
-
26.04 bug dark-mode Easy Hack impress symptom: visual
难度 2/5 1-3 小时 新手友好度 68/100
CollaboraOnline/online#16421 · 1 条评论 ·
-
enhancement
难度 2/5 1-3 小时 新手友好度 63/100
telegramdesktop/tdesktop#31653 ·
维护者通常 1 天内回复
-
area/cs bug
难度 2/5 1-3 小时 新手友好度 82/100
ydb-platform/ydb#59881 · 1 条评论 ·
维护者通常 1 天内回复