Metal cache admission can exceed the configured cache limit
Maintainers usually reply within 3 days
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 25/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Stale
- Tech stack
- cpp
- Domain
- backend, performance
Research direction
Start by locating MetalAllocator::free() and the cache limit checks described in the issue. Reproduce the oversized and mixed allocation cases with the allocator tests, then inspect the linked pull request #4842 for the two-file fix and regression coverage. Done means freed buffers respect the configured limit, including zero, and the reported reuse cases are covered.
Written by the indexing model from the issue text.
Description
MetalAllocator::free() checks the current cache size before admitting the incoming buffer, without including that buffer's length. This can exceed set_cache_limit() as soon as a live allocation is released.
On Apple M1 Max, using current main 9f806b3606eaa9f7c807238f6bd465185cd94d6d:
- Set the cache limit to 1 MiB and clear it.
- Allocate and release a 2 MiB buffer:
get_cache_memory()reports 2,097,152 bytes. - Clear it, hold a 512 KiB allocation while allocating/releasing a 768 KiB allocation, then release the 512 KiB allocation: the cache reaches 1,310,720 bytes with a 1,048,576-byte limit.
No later allocation is made before inspecting each overshoot. The checks use allocator::malloc directly, so shader execution and asynchronous command completion are not involved in the reproduction.
Expected: an incoming buffer larger than the allowance should not enter the cache. A smaller incoming buffer should be admitted only after enough older cached bytes are released. Zero should continue to disable reuse. This concerns the freed-buffer cache, not a limit on active allocations or total process memory.
I have a small two-file fix and regression test covering oversized admission, mixed shapes, recent-shape reuse and zero allowance.
- Dominant language
- C++
- Stars
- 29.1k
- Forks
- 2.4k
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 103
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from ml-explore/mlx
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
ml-explore/mlx#4832 · 1 comment ·
Maintainers usually reply within 3 days
-
[BUG] `mx.expand_dims` docs: the Args section names `axes`, but the parameter is `axis`Possibly taken @Redempt1onzzZZ claimed this 3 days ago. Open
Difficulty 1/5 Under an hour Newbie friendliness 88/100
ml-explore/mlx#4829 ·
Maintainers usually reply within 3 days
-
[BUG] `mx.qqmm` docs: the Args section names `global_scale`, but the parameter is `global_scale_x`Possibly taken @XXXXRT666 claimed this 5 days ago. Open
Difficulty 1/5 Under an hour Newbie friendliness 92/100
ml-explore/mlx#4815 ·
Maintainers usually reply within 3 days
-
[BUG] FP4 quantization on CPU rounds ties toward zero while Metal rounds to evenMay be free again A pull request for this issue was closed without being merged. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
ml-explore/mlx#4753 ·
Maintainers usually reply within 3 days
-
Difficulty 4/5 3-5 days Newbie friendliness 43/100
ml-explore/mlx#4843 ·
Maintainers usually reply within 3 days
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
arvidn/libtorrent#9126 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Sunrisepeak/mcpp-language-server#80 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
xiaoyifang/goldendict-ng#3148 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 91/100
triton-inference-server/server#9069 ·
Maintainers usually reply within 1 day