JuliaDynamics/DrWatson.jl

Usage together with Distributed can cause precompile issues

Open

#270 opened on Jul 23, 2021

View on GitHub
 (1 comment) (2 reactions) (0 assignees)Julia (915 stars) (94 forks)batch import
help wantedproject-setup

Description

As mentioned already in https://github.com/JuliaDynamics/DrWatson.jl/issues/269, using DrWatson together with the Distributed package can cause errors where many remote worker processes try to precompile packages at the same time. This either causes the machine to run out of memory or raise issues with the compilecache.

MWE for the error (now using a pure Julia package):

using Distributed
addprocs(8)
@everywhere using DrWatson
@everywhere @quickactivate "TestEnv" 

@everywhere begin
    using Distributions
end

Slightly altering the order of imports solves the issue. This works:

using Distributed
addprocs(8)
@everywhere using DrWatson

@everywhere begin
    @quickactivate "TestEnv" 
    using Distributions
end

The issue seems indeed to be related to DrWatson, because running

using Distributed
addprocs(8)
@everywhere import Pkg
@everywhere Pkg.activate(".") 

@everywhere begin
    using Distributions
end

also poses no problem.

I am really interested to know the reasons for this.

EDIT: Last example corrected to better match the issue case.

Contributor guide

Tech stack
None
Domain
backendperformance
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
basic JuliaDistributed package
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40
Research direction
This issue presents a bug where using DrWatson with Distributed causes precompile errors when @quickactivate is used inside @everywhere. The workaround is to separate the calls. Investigate the interaction between DrWatson's @quickactivate macro and Julia's precompilation system, focusing on the order of operations and potential race conditions when multiple workers precompile simultaneously. Look at the code in src/quickactivate.jl and the precompilation hooks in DrWatson.