JuliaDynamics/DrWatson.jl

Usage together with Distributed can cause precompile issues

开放

#270 创建于 2021年7月23日

 (1 条评论) (2 个反应) (0 位负责人)Julia (94 个派生)batch import
help wantedproject-setup

仓库指标

星标
 (915 个星标)
PR 合并指标
 (30 天内没有已合并 PR)

描述

As mentioned already in https://github.com/JuliaDynamics/DrWatson.jl/issues/269, using DrWatson together with the Distributed package can cause errors where many remote worker processes try to precompile packages at the same time. This either causes the machine to run out of memory or raise issues with the compilecache.

MWE for the error (now using a pure Julia package):

using Distributed
addprocs(8)
@everywhere using DrWatson
@everywhere @quickactivate "TestEnv" 

@everywhere begin
    using Distributions
end

Slightly altering the order of imports solves the issue. This works:

using Distributed
addprocs(8)
@everywhere using DrWatson

@everywhere begin
    @quickactivate "TestEnv" 
    using Distributions
end

The issue seems indeed to be related to DrWatson, because running

using Distributed
addprocs(8)
@everywhere import Pkg
@everywhere Pkg.activate(".") 

@everywhere begin
    using Distributions
end

also poses no problem.

I am really interested to know the reasons for this.

EDIT: Last example corrected to better match the issue case.

贡献者指南