Option for filtering by CUDNN_DETERMINISTIC in `cudnnConvolutionAlgoPerfChoose`
#938 opened on May 30, 2021
Repository metrics
- Stars
- (1,408 stars)
- PR merge metrics
- (Avg merge 5d 5h) (16 merged PRs in 30d)
Description
Is your feature request related to a problem? Please describe.
Following from https://discourse.julialang.org/t/flux-reproducibility-of-gpu-experiments/62092, there is no way to guarantee for users of e.g. NNlibCUDA to ensure convolution operations only use deterministic algorithms
Describe the solution you'd like
Something along the lines of https://github.com/pytorch/pytorch/blob/6c70cbedb6102da08fe91186d40a41b50991681d/aten/src/ATen/native/cudnn/Conv_v7.cpp#L219-L252. Whether this would need to be plumbed through higher-level functions, set as a global option or exposed through a context manager is left for debate.
Describe alternatives you've considered
The only solution now is seems to be pirating cudnnConvolutionForwardAD such that it doesn't use cudnnConvolutionFwdAlgoPerf?
Additional context
https://docs.nvidia.com/deeplearning/cudnn/developer-guide/index.html#reproducibility states:
...the following routines do not guarantee reproducibility because they use atomic operations:
- cudnnConvolutionBackwardFilter when CUDNN_CONVOLUTION_BWD_FILTER_ALGO_0 or CUDNN_CONVOLUTION_BWD_FILTER_ALGO_3 is used
- cudnnConvolutionBackwardData when CUDNN_CONVOLUTION_BWD_DATA_ALGO_0 is used
So I have no clue if/how often these show up in practice in cudnnConvolution*AlgoPerf, but I assume they must if users are seeing non-deterministic results?