Provide heuristic-based fallback mode for conv algo search
#2 108 ouverte le 3 oct. 2023
Métriques du dépôt
- Stars
- (1 408 stars)
- Métriques de merge PR
- (Merge moyen 5j 5h) (16 PRs mergées en 30 j)
Description
Is your feature request related to a problem? Please describe.
Startup time caused by repeated algorithm searches has been a long-standing issue when running CNNs on GPU in Julia. It would be nice to have a way to bypass the overhead of such searches while still maintaining some semblance of performance.
Describe the solution you'd like
By default, PyTorch does not perform an algorithm search and instead uses some default fallback. Ref. the cuDNN v7 and v8 API using code paths which handle this.
Describe alternatives you've considered
The main alternative would be caching saved configurations to disk for further use as mentioned in https://github.com/JuliaGPU/CUDA.jl/issues/1947. However, the specifics of how to persist the cache may take some time and effort to figure out. Given that PyTorch has yet to implement something similar on their end, I assume the design would be non-trivial.
Additional context
Somewhat interestingly, MIOpen provides a built-in caching mechanism for this purpose. Unfortunately cuDNN does not appear to, but perhaps they will feel pressure to add something similar eventually. In either case, having a fallback which doesn't require cache priming feels like a good idea.