TaskResources release loop aborts on the first failing release, leaking the rest

Open
#13,106 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
58/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
scala

Research direction

Start at TaskResourceRegistry.releaseAll and trace the completion listener that calls RESOURCE_REGISTRIES.remove(context). Inspect how priorityToResourcesMapping, resources, ColumnarBatch release(), jniWrapper.stop, and Runtime cleanup behave when release() throws. Done means every resource is attempted, registry state is cleared, and cleanup continues after an exception without leaking lower-priority resources.

Written by the indexing model from the issue text.

Description

TaskResourceRegistry.releaseAll releases task resources highest-priority first (100 → 50 → 30), in reverse registration order. If one resource's release() throws, the foreach aborts: every remaining lower-priority resource is skipped, and priorityToResourcesMapping / resources are never cleared. The completion listener also then skips RESOURCE_REGISTRIES.remove(context).

A throwing release() is reachable on task failure/teardown. Priority-100 recyclers close native ColumnarBatch payloads and priority-50 shuffle readers call jniWrapper.stop, both of which surface JNI errors as exceptions. When one of those throws, the priority-30 Runtime (which owns the native memory manager and native threads) never releases, so its off-heap memory leaks. Across repeatedly-failing tasks on a long-lived executor this compounds into an OOM or container kill.

Dominant language
Scala
Stars
1.6k
Forks
679
Avg merge
4d 16h
Merged PRs (30d)
84

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/gluten

All issues in apache/gluten

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.