Memory leakage upon repeated training
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- julia
- Domain
- machine-learning
Research direction
Start by running the Julia MWE with DecisionTree, focusing on repeated calls to build_forest and apply_forest in run_forests. Compare memory behavior with run_something_else and investigate why trees are not reclaimed after run_forests returns. Done means identifying and correcting the reported memory growth while preserving the stable behavior of the second loop.
Written by the indexing model from the issue text.
Description
Hi, I've been creating some random forest regressors lately and I've noticed high memory usage during hyperparameter tuning. It turns out that there is some memory leakage in the package. For some reason Julia does not delete the trees when they become unreachable.
Following is a MWE: after finishing run_forests, some memory should be reclaimed but it doesn't happen and memory usage increases. When running the second loop however, memory usage stays constant.
using DecisionTree
function run_forests(features, labels)
forest = build_forest(labels, features)
labels .+= apply_forest(forest, features)
labels ./= 2
end
function run_something_else(features, labels)
C = repeat(features, inner=(2,2))
labels ./= vec(sum(C, dims=2))[1:length(labels)]
end
const features = rand(10_000, 10)
const labels = sum(features, dims=2) |> vec
# notice memory consumption increases every couple of iterations
for i = 1:1_000
run_forests(features, labels)
@info "Iteration $i current memory used" Sys.maxrss()
end
# notice memory consumption does not increase every couple of iterations
for i = 1:1_000
run_something_else(features, labels)
@info "Iteration $i current memory used" Sys.maxrss()
end
Any idea what might cause this?
- Dominant language
- Julia
- Stars
- 365
- Forks
- 100
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from JuliaAI/DecisionTree.jl
-
bug
Difficulty 3/5 1-2 days Newbie friendliness 45/100
JuliaAI/DecisionTree.jl#234 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 35/100
JuliaAI/DecisionTree.jl#225 · 2 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 20/100
JuliaAI/DecisionTree.jl#222 · 1 comment ·
-
good first issue
Difficulty 3/5 1-2 days Newbie friendliness 35/100
JuliaAI/DecisionTree.jl#217 · 4 comments ·
-
documentation
Difficulty 4/5 3-5 days Newbie friendliness 35/100
JuliaAI/DecisionTree.jl#215 ·
All issues in JuliaAI/DecisionTree.jl
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 79/100
Sienna-Platform/PowerSimulations.jl#1737 ·
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
FluxML/Optimisers.jl#254 ·
-
Minor: floodplain warning percentage, scalar output error message and a never-true identity checkOpenbug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
JuliaAI/MLJTransforms.jl#101 ·
Maintainers usually reply within 1 day
-
@contents blocks fail to render on Julia 1.14Possibly taken @sbfnk claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
LuxDL/DocumenterVitepress.jl#410 ·