Option for filtering by CUDNN_DETERMINISTIC in `cudnnConvolutionAlgoPerfChoose`
#938 geöffnet am 30. Mai 2021
Repository-Metriken
- Stars
- (1.408 Stars)
- PR-Merge-Metriken
- (Durchschn. Merge 5T 5h) (16 gemergte PRs in 30 T)
Beschreibung
Is your feature request related to a problem? Please describe.
Following from https://discourse.julialang.org/t/flux-reproducibility-of-gpu-experiments/62092, there is no way to guarantee for users of e.g. NNlibCUDA to ensure convolution operations only use deterministic algorithms
Describe the solution you'd like
Something along the lines of https://github.com/pytorch/pytorch/blob/6c70cbedb6102da08fe91186d40a41b50991681d/aten/src/ATen/native/cudnn/Conv_v7.cpp#L219-L252. Whether this would need to be plumbed through higher-level functions, set as a global option or exposed through a context manager is left for debate.
Describe alternatives you've considered
The only solution now is seems to be pirating cudnnConvolutionForwardAD such that it doesn't use cudnnConvolutionFwdAlgoPerf?
Additional context
https://docs.nvidia.com/deeplearning/cudnn/developer-guide/index.html#reproducibility states:
...the following routines do not guarantee reproducibility because they use atomic operations:
- cudnnConvolutionBackwardFilter when CUDNN_CONVOLUTION_BWD_FILTER_ALGO_0 or CUDNN_CONVOLUTION_BWD_FILTER_ALGO_3 is used
- cudnnConvolutionBackwardData when CUDNN_CONVOLUTION_BWD_DATA_ALGO_0 is used
So I have no clue if/how often these show up in practice in cudnnConvolution*AlgoPerf, but I assume they must if users are seeing non-deterministic results?