bloomberg/memray

Update the C++ FileSink to fall back to write() if memory mapping isn't supported

Open

#748 opened on Apr 11, 2025

View on GitHub
 (5 comments) (0 reactions) (1 assignee)Python (11,796 stars) (383 forks)batch import
enhancementgood first issue

Description

For the sake of performance, memray writes to capture files by mapping the capture file into memory and then writing to it as an in-memory array that is eventually synchronized to the filesystem. (See FileSink in sink.cpp / sink.h)

This is fast, but some (uncommon and non-standard) filesystems don't support it.

We should detect if the operations that we need (posix_fallocate and mmap) aren't supported by the filesystem that the capture file is being written to, and if so log a warning and fall back to a slower but more portable path that simply uses write() to append each new chunk of data to the capture file. In order to test this, we'll also need to add a way to force the slower path to be used even if posix_fallocate and mmap are supported. This should probably just be a MEMRAY_NO_MMAP=1 environment variable, or something like that.

Contributor guide

Tech stack
cpppython
Domain
performance
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
blocked
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
C++File I/OMemory mappingMemray internals
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40
Research direction
Investigate sink.cpp and sink.h to understand the current FileSink implementation. Detect at runtime whether posix fallocate and mmap are supported by the filesystem, possibly using sysconf or checking errno. If not supported, log a warning and fall back to using write() to append data. Add a MEMRAY NO MMAP environment variable to force the fallback path for testing. Look at existing test files for file writing to ensure proper testing.
Update the C++ FileSink to fall back to write() if memory mapping isn't supported · bloomberg/memray#748 | Good First Issue