Reuse executor for repeated prediction with different checkpoints
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- julia
- Domain
- machine-learning
Research direction
Start with the mx.predict path and the arch.pred_exec lifecycle shown in the issue. Trace how the executor is created and finalized when arch.arg_params and arch.aux_params change; done means repeated checkpoint predictions reuse the executor without the explicit finalization workaround or the reported memory growth.
Written by the indexing model from the issue text.
Description
Testing the different checkpoints of a training run requires loading checkpoints and predictions runs in a tight loop.
net = mx.load(archfile, mx.SymbolicNode)
arch = mx.FeedForward(net, context=mx.gpu())
data = ...
for wfile in weights
saved_dict = mx.load(wfile, mx.NDArray)
arg_params = Dict{Base.Symbol, mx.NDArray}()
aux_params = Dict{Base.Symbol, mx.NDArray}()
for (k,v) in saved_dict
tp, name = split(string(k), ':')
name = symbol(name)
if tp == "arg"
arg_params[name] = v
else
aux_params[name] = v
end
end
arch.arg_params = arg_params
arch.aux_params = aux_params
pred = mx.predict(arch, data)
# For memory reclaim, eagerly finalize self.pred_exec.handle
finalize(arch.pred_exec.handle)
arch.pred_exec=nothing
end
Without these final two lines this easily runs out of memory for big models/batch_sizes, because the executor is not gc'd yet and we are creating a new one asking for more memory.
If we could reuse the previous executor that problem would be alleviated.
- Dominant language
- No language data
- Stars
- 370
- Forks
- 63
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from dmlc/MXNet.jl
-
Import fail MXNET.jlOpen
Difficulty 3/5 1-2 days Newbie friendliness 28/100
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
Similar issues
-
good first issue hardware help wanted no-box
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
Maintainers usually reply within 1 day
-
pnnx Python convert ignores fp16=False with input_shapesPossibly taken @ktz03 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Tencent/ncnn#7183 · 1 reaction ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
YuE2 / MiniMax Music 3 int8 text encoders crash on MPS with aten::_int_mm (missing disabled quant formats)Possibly taken @cunderw claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
UKGovernmentBEIS/optstop#10 ·