JuliaGPU/CUDA.jl

Don't sync during graph record

Open

#1,229 opened on Nov 3, 2021

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Julia (274 forks)batch import
buggood first issue

Repository metrics

Stars
 (1,408 stars)
PR merge metrics
 (Avg merge 5d 5h) (16 merged PRs in 30d)

Description

We eagerly sync during copyto to avoid blocking, but that doesn't work when recording a graph:

julia> @captured CuArray([1])
ERROR: CUDA error: operation failed due to a previous error during capture (code 901, ERROR_STREAM_CAPTURE_INVALIDATED)
Stacktrace:
 [1] throw_api_error(res::CUDA.cudaError_enum)
   @ CUDA ~/Julia/pkg/CUDA/lib/cudadrv/error.jl:91
 [2] capture(f::var"#11#13"; flags::CUDA.CUstreamCaptureMode_enum, throw_error::Bool)
   @ CUDA ~/Julia/pkg/CUDA/lib/cudadrv/graph.jl:44
 [3] top-level scope
   @ ~/Julia/pkg/CUDA/lib/cudadrv/graph.jl:213
 [4] top-level scope
   @ ~/Julia/pkg/CUDA/src/initialization.jl:52

caused by: CUDA error: operation not permitted when stream is capturing (code 900, ERROR_STREAM_CAPTURE_UNSUPPORTED)
Stacktrace:
  [1] throw_api_error(res::CUDA.cudaError_enum)
    @ CUDA ~/Julia/pkg/CUDA/lib/cudadrv/error.jl:91
  [2] isdone
    @ ~/Julia/pkg/CUDA/lib/cudadrv/stream.jl:107 [inlined]
  [3] nonblocking_synchronize
    @ ~/Julia/pkg/CUDA/lib/cudadrv/stream.jl:137 [inlined]
  [4] macro expansion
    @ ~/Julia/pkg/CUDA/src/array.jl:388 [inlined]
  [5] macro expansion
    @ ~/Julia/pkg/CUDA/src/state.jl:182 [inlined]
  [6] unsafe_copyto!(dest::CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, doffs::Int64, src::Vector{Int64}, soffs::Int64, n::Int64)
    @ CUDA ~/Julia/pkg/CUDA/src/array.jl:384
  [7] copyto!
    @ ~/Julia/pkg/CUDA/src/array.jl:342 [inlined]

Contributor guide