JuliaDynamics/DrWatson.jl

Usage together with Distributed can cause precompile issues

Open

#270 创建于 2021年7月23日

在 GitHub 查看
 (1 评论) (2 反应) (0 负责人)Julia (915 star) (94 fork)batch import
help wantedproject-setup

描述

As mentioned already in https://github.com/JuliaDynamics/DrWatson.jl/issues/269, using DrWatson together with the Distributed package can cause errors where many remote worker processes try to precompile packages at the same time. This either causes the machine to run out of memory or raise issues with the compilecache.

MWE for the error (now using a pure Julia package):

using Distributed
addprocs(8)
@everywhere using DrWatson
@everywhere @quickactivate "TestEnv" 

@everywhere begin
    using Distributions
end

Slightly altering the order of imports solves the issue. This works:

using Distributed
addprocs(8)
@everywhere using DrWatson

@everywhere begin
    @quickactivate "TestEnv" 
    using Distributions
end

The issue seems indeed to be related to DrWatson, because running

using Distributed
addprocs(8)
@everywhere import Pkg
@everywhere Pkg.activate(".") 

@everywhere begin
    using Distributions
end

also poses no problem.

I am really interested to know the reasons for this.

EDIT: Last example corrected to better match the issue case.

贡献者指南