JuliaDynamics/DrWatson.jl

Usage together with Distributed can cause precompile issues

Open

#270 aperta il 23 lug 2021

Vedi su GitHub
 (1 commento) (2 reazioni) (0 assegnatari)Julia (94 fork)batch import
help wantedproject-setup

Metriche repository

Star
 (915 star)
Metriche merge PR
 (Nessuna PR mergiata in 30 g)

Descrizione

As mentioned already in https://github.com/JuliaDynamics/DrWatson.jl/issues/269, using DrWatson together with the Distributed package can cause errors where many remote worker processes try to precompile packages at the same time. This either causes the machine to run out of memory or raise issues with the compilecache.

MWE for the error (now using a pure Julia package):

using Distributed
addprocs(8)
@everywhere using DrWatson
@everywhere @quickactivate "TestEnv" 

@everywhere begin
    using Distributions
end

Slightly altering the order of imports solves the issue. This works:

using Distributed
addprocs(8)
@everywhere using DrWatson

@everywhere begin
    @quickactivate "TestEnv" 
    using Distributions
end

The issue seems indeed to be related to DrWatson, because running

using Distributed
addprocs(8)
@everywhere import Pkg
@everywhere Pkg.activate(".") 

@everywhere begin
    using Distributions
end

also poses no problem.

I am really interested to know the reasons for this.

EDIT: Last example corrected to better match the issue case.

Guida contributor