Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[DISCUSS][CORE] Should managed off-heap memory take the container limit into account?

Aperta
  • Qualcosa non va?
#57,530 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Da chiarire
Stato di attività
Tranquilla
Stack tecnologico
scala
Ambito
distributed-systems, performance

Direzione di ricerca

Inizia con UnifiedMemoryManager e il percorso di spill esistente di MemoryConsumer, quindi esamina le letture cgroup v1/v2 del prototipo e le ipotesi sul budget JVM/native. Un risultato utile sarebbe una direzione approvata dai maintainer — configurazione dell’executor-overhead, un limite o un avviso all’avvio, oppure feedback a runtime — seguita da un reproducer mirato e da test.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

I have been looking at how Spark managed off-heap memory behaves in a memory-limited container.
One thing that seems easy to misconfigure is that these three limits are independent:

container limit:                 1500 MiB
JVM max heap:                    1024 MiB
spark.memory.offHeap.size:       2048 MiB

spark.memory.offHeap.size is only an upper bound used by Spark's memory manager, so the example
above does not fail immediately. The problem is that UnifiedMemoryManager does not know about the
1500 MiB process limit. If heap, managed off-heap memory, direct buffers, thread stacks, and other
native allocations grow at the same time, the container can kill the executor before Spark gets a
chance to spill.

In that case there may be no useful Java OOM. From outside the container it can just look like:

exit code 137
OOMKilled=true

Using JVM heap occupancy as the feedback signal does not seem right here. It cannot see off-heap
or other native memory, and it moves with GC. The cgroup usage and limit describe the boundary that
actually kills the process.

I made a small prototype that reads cgroup v1/v2 values and uses them to put a lower admission
limit on Spark-managed off-heap memory. At startup the rough budget is:

cgroup limit - JVM max heap - native reserve

The admission limit can then be reduced when total cgroup usage is high. This still relies on the
existing MemoryConsumer and spill code; it does not add another eviction mechanism.

There are some non-obvious details. In particular, setting the limit to zero is not safe because a
task may fail to allocate its first sorter page and throw SparkOutOfMemoryError instead of
spilling. Also, a fixed native reserve is probably too simplistic across different collectors,
core counts, and deployment modes.

Before spending more time on the prototype, I would like to check the expected direction:

  1. Should Spark use the container limit when admitting managed off-heap allocations, or is this
    expected to be handled entirely by executor memory-overhead configuration?
  2. If Spark should handle it, would a startup warning/static cap be preferable to runtime
    cgroup-pressure feedback?

If this belongs in Spark Core, I can follow up with a focused reproducer and tests for the approach
maintainers prefer.

Lingua principale
Scala
Stelle
44.2k
Fork
29.5k
Merge medio
8h 43m
PR unite (30g)
1

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/spark

Tutte le issue di apache/spark

Issue simili

Altre issue su Scala

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.