Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Memory usage grows with each `SCAN` call once objects are evicted to disk

Open
  • Is something wrong?
#2,330 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
csharp
Domain
databases

Research direction

Start by reproducing the leak with the attached ScanRetentionRepro program against a storage-tier Garnet server, then trace the SCAN implementation and its handling of objects evicted to disk. Compare retained memory across full scans with different COUNT values after a full GC. Done means repeated scans no longer cause retained memory growth; the issue does not identify a source file or test.

Written by the indexing model from the issue text.

Description

Describe the bug

When a store holds an object (here a list) that has been evicted to disk, a full SCAN iteration that takes several calls leaves about that object's size of memory in use per call. The memory stays in use after a full garbage collection, and repeated scans add to it. Nothing was written or changed during SCAN

Console App which reproduces the issue: ScanRetentionRepro.zip

With a 42.3 MB list on disk and 2,502 keys, the attached program measured live managed memory after a full GC:

Full SCAN iteration Calls Live managed memory change
COUNT 100000 1 +16.1 MB*
COUNT 1000 3 +84.5 MB
COUNT 100 26 +1,014.8 MB

* A one-time increase on the first scan that returns every key; further single-call scans add nothing.

The scans ran one after another, after one warm-up scan. Five seconds after the last one, after a full GC, the process still held 1,115.6 MB more than before the scans.

Common clients use small COUNT values by default (redis-cli --scan uses 10, StackExchange.Redis's KeysAsync pages 250 at a time), so a full iteration typically takes many calls.

The same memory usage growth also reproduces outside a process hosting Garnet. On a standalone garnet-server 2.2.5 with 3,445 keys. Reading gc_heap_bytes from INFO MEMORY after DEBUG FORCEGC:

  • A 4-call COUNT 1000 iteration added 129.0 MB.
  • A 35-call COUNT 100 iteration added 1,424.7 MB.
  • 60 seconds later, and after 10,000 more small string SETs, usage was unchanged.
GarnetServer.exe --port 6390 --bind 127.0.0.1 --memory 64m --page 16m --index 1m --storage-tier
  --logdir <dir>\log --checkpointdir <dir>\checkpoints --enable-debug-command local
Steps to reproduce the bug

A self-contained program is attached: ScanRetentionRepro.zip: Program.cs and ScanRetentionRepro.csproj). It hosts Garnet in-process, uses StackExchange.Redis, and runs in about 15 seconds:

dotnet run -c Release

It does the following:

  1. Start Garnet with: 
    --memory 64m --page 16m --index 1m --storage-tier --logdir <dir> --checkpointdir <dir>
  2. RPUSH list 7,000 ten-byte elements at a time, 72 times (504,000 elements).
  3. Write small string keys (SET filler:<n> <32 bytes>) in batches of 128, with one RPUSH wake x after each batch, until Log.CurrentHeapSizeBytes in INFO STORE drops to 0 (the list has been evicted). The RPUSH is needed because writes that don't change Log.CurrentHeapSizeBytes don't trigger eviction. Then write more filler keys until there are 2,500, for 2,502 keys in total with list and wake.
  4. Without writing anything else, run full SCAN iterations with different COUNT values. After each, measure the process's live managed memory after a full GC (GC.GetTotalMemory(true), which is why the program hosts Garnet itself).

Output on 2.2.5 (identical on 2.2.4):

Garnet 2.2.5.0, in-process: --memory 64m --page 16m --index 1m --storage-tier

Setup
  1. RPUSH list, 504,000 elements (42.3 MB as counted by Garnet)
  2. SET small keys until the list is evicted to disk
  3. keys in the database: 2,502

Full SCAN iterations, with no writes in between
                                                             live memory
  run      command                     calls  keys returned      change*  notes
  warm-up  SCAN <cursor> COUNT 100000      1     65 / 2,502      +4.2 MB  excluded from the result; misses keys, a separate issue
  1        SCAN <cursor> COUNT 100000      1  2,502 / 2,502     +16.1 MB
  2        SCAN <cursor> COUNT 1000        3  2,502 / 2,502     +84.5 MB
  3        SCAN <cursor> COUNT 100        26  2,502 / 2,502   +1014.8 MB
  Each run sends the command from cursor 0 until the returned cursor is 0; calls = commands sent.
  * change in the process's live managed memory after a full GC (GC.GetTotalMemory(true)); the list is 42.3 MB

Result
  RETAINED: 5 seconds after the last scan, after a full GC, live managed memory was still 1115.6 MB above its level
            before runs 1-3. The scans wrote nothing; the only large object, the list, is 42.3 MB.

The program exits with code 2 when the growth reproduces.

The same scans also increase Log.CurrentHeapSizeBytes in INFO STORE, and the warm-up scan returns only 65 of 2,502 keys. Those are separate problems; I'll report them in their own issues and link them here.

The same results occur with AOF on (--aof --aof-commit-wait --aof-size-limit 256m), --compaction-type Lookup --compaction-max-segments 4, and --max-databases 2 with the data in DB 1.

Expected behavior

A SCAN iteration does not leave memory in use after it completes, whatever its COUNT.

Screenshots

Not applicable; console output is included above.

Release version

No response

IDE

No response

OS version

No response

Additional context

No response

Dominant language
C#
Stars
12.1k
Forks
716
Avg merge
2d 2h
Merged PRs (30d)
65

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/garnet

All issues in microsoft/garnet

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.