[BUG] sparse matrix multiplication causes access violation

Creating a sparse matrix and then multiplying with same or other sparse matrix or with another matrix causes crash and access violation.

Description

• Did you build ArrayFire yourself or did you use the official installers:
  use official installer
• Which backend is experiencing this issue? (CPU, CUDA, OpenCL)
  CUDA
• Do you have a workaround?
  NO
• Can the bug be reproduced reliably on your system?
  YES
• A clear and concise description of what you expected to happen.
• Run your executable with AF_TRACE=all and AF_PRINT_ERRORS=1 environment
  variables set.
• Screenshot or terminal output of the results
  see below

Reproducible Code and/or Steps

I am running the following code n GOOGLE tests, However, I am having trouble with sparse matrices in other places as well. Similar nature.

TEST(TestSuiteName, TestArrayFire)
{
float v[] = {5, 8, 3, 6};
int r[] = {0, 0, 2, 3, 4};
int c[] = {0, 1, 2, 1};
const int M = 4, N = 4, nnz = 4;
array vals = array(dim4(nnz), v);
array row_ptr = array(dim4(M + 1), r);
array col_idx = array(dim4(nnz), c);

  af_print(vals);
  af_print(row_ptr);
  af_print(col_idx);

  // Create sparse array (CSR) from af::arrays containing values,
  // row pointers, and column indices.
  array sparse = af::sparse(M, N, vals, row_ptr, col_idx, AF_STORAGE_CSR);
  af_print(sparse);

  auto res = sparse * sparse;
  af_print(res);

}

StackTrace when debugging:

af.dll!af_get_last_error(char * * str, __int64 * len) Line 46 (f:\buildbot\worker\win10-cuda-installer\build\src\api\unified\error.cpp:46)
af.dll!af::operator*(const af::array & lhs, const af::array & rhs) Line 939 (f:\buildbot\worker\win10-cuda-installer\build\src\api\cpp\array.cpp:939)

Output:

Loaded 'C:\Windows\System32\dcomp.dll'.
[platform][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\opencl\device_manager.cpp(361) ] Default device: 0
[unified][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\api\unified\symbol_manager.cpp(153) ] Device Count: 4.
[unified][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\api\unified\symbol_manager.cpp(143) ] Attempting: Default System Paths
Loaded 'C:\Programs\ArratFire\v3\lib\afcuda.dll'. Symbols loaded.
Loaded 'C:\Windows\System32\nvcuda.dll'.
Loaded 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2\bin\cublas64_12.dll'. Module was built without symbols.
Loaded 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2\bin\cufft64_11.dll'. Module was built without symbols.
Loaded 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2\bin\cusolver64_11.dll'. Module was built without symbols.
2
Loaded 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2\bin\cusparse64_12.dll'. Module was built without symbols.
Unloaded 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2\bin\cusparse64_12.dll'.
Loaded 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2\bin\nvrtc64_120_0.dll'. Module was built without symbols.
Loaded 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2\bin\cublasLt64_12.dll'. Module was built without symbols.
Loaded 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2\bin\nvJitLink_120_0.dll'. Module was built without symbols.
[unified][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\api\unified\symbol_manager.cpp(146) ] Found: afcuda.dll
Loaded 'C:\Windows\System32\DriverStore\FileRepository\nvdmi.inf_amd64_893ed8ff453738db\nvcuda64.dll'.
[unified][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\api\unified\symbol_manager.cpp(153) ] Device Count: 1.
[unified][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\api\unified\symbol_manager.cpp(208) ] AF_DEFAULT_BACKEND: cuda
[platform][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\common\DependencyModule.cpp(101) ] Attempting to load: forge.dll
[platform][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\common\DependencyModule.cpp(104) ] Found: forge.dll
[platform][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\device_manager.cpp(494) ] CUDA Driver supports up to CUDA 12.2.0 ArrayFire CUDA Runtime 12.0.0
[platform][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\device_manager.cpp(479) ] CUDA driver version(12.2.0) not part of the CudaToDriverVersion array. Please create an issue or a pull request on the ArrayFire repository to update the CudaToDriverVersion variable with this version of the CUDA runtime.

[platform][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\device_manager.cpp(562) ] Found 1 CUDA devices
[platform][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\device_manager.cpp(590) ] Found device: NVIDIA GeForce GTX 1050 Ti with Max-Q Design (sm_61) (4 GB | ~2076.416015625 GFLOPs | 6 SMs)
[platform][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\device_manager.cpp(625) ] AF_CUDA_DEFAULT_DEVICE:
[platform][1688279121][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\device_manager.cpp(644) ] Default device: 0(NVIDIA GeForce GTX 1050 Ti with Max-Q Design)
[mem][1688279121][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\common\DefaultMemoryManager.cpp(128) ] memory[0].max_bytes: 3 GB
[mem][1688279121][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\memory.cpp(157) ] nativeAlloc: 1 KB 0x1001000000
[mem][1688279121][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\memory.cpp(157) ] nativeAlloc: 1 KB 0x1001000400
[mem][1688279121][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\memory.cpp(157) ] nativeAlloc: 1 KB 0x1001000800
als
[4 1 1 1]
[mem][1688279137][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\memory.cpp(157) ] nativeAlloc: 1 KB 0x1001000c00
[jit][1688279137][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\compile_module.cpp(466) ] {291210446400920389 : loaded from C:\Users\rohrb\AppData\Local\Temp\ArrayFire\KER291210446400920389_CU_61_AF_38.bin for NVIDIA GeForce GTX 1050 Ti with Max-Q Design }
[kernel][1688279137][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\Kernel.hpp(42) ] Launching arrayfire::cuda::transpose<float,false,false>: Blocks: [1, 1, 1] Threads: [32, 8, 1] Shared Memory: 0
5.0000
8.0000
3.0000
6.0000

row_ptr
[5 1 1 1]
[jit][1688279139][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\compile_module.cpp(466) ] {9963207849209086996 : loaded from C:\Users\rohrb\AppData\Local\Temp\ArrayFire\KER9963207849209086996_CU_61_AF_38.bin for NVIDIA GeForce GTX 1050 Ti with Max-Q Design }
[kernel][1688279139][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\Kernel.hpp(42) ] Launching arrayfire::cuda::transpose<int,false,false>: Blocks: [1, 1, 1] Threads: [32, 8, 1] Shared Memory: 0
2
0
2
3
4

col_idx
[4 1 1 1]
[kernel][1688279147][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\Kernel.hpp(42) ] Launching arrayfire::cuda::transpose<int,false,false>: Blocks: [1, 1, 1] Threads: [32, 8, 1] Shared Memory: 0
0
1
2
1

sparse
Storage Format : AF_STORAGE_CSR
[4 4 1 1]
sparse: Values
[4 1 1 1]
[kernel][1688279152][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\Kernel.hpp(42) ] Launching arrayfire::cuda::transpose<float,false,false>: Blocks: [1, 1, 1] Threads: [32, 8, 1] Shared Memory: 0
5.0000
8.0000
3.0000
6.0000

sparse: RowIdx
[5 1 1 1]
[kernel][1688279152][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\Kernel.hpp(42) ] Launching arrayfire::cuda::transpose<int,false,false>: Blocks: [1, 1, 1] Threads: [32, 8, 1] Shared Memory: 0
2
0
2
3
4

sparse: ColIdx
[4 1 1 1]
[kernel][1688279152][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\Kernel.hpp(42) ] Launching arrayfire::cuda::transpose<int,false,false>: Blocks: [1, 1, 1] Threads: [32, 8, 1] Shared Memory: 0
0
1
2
1

Exception thrown at 0x00007FFF04ADADCC (af.dll) in runTests.exe: 0xC0000005: Access violation reading location 0xFFFFFFFFFFFFFFFF.
unknown file: error: SEH exception with code 0xc0000005 thrown in the test body.

unknown file: error: SEH exception with code 0xc0000005 thrown in the test body.

System Information

Please provide the following information:

1. ArrayFire version
   ArrayFire v3.8.3

2. Devices installed on the system
   2 Graphics Cards 1 CPU
   [platform][1688279118][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\common\DependencyModule.cpp(104) ] Found: forge.dll
   [platform][1688279118][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\opencl\device_manager.cpp(217) ] Found 5 OpenCL platforms
   [platform][1688279118][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\opencl\device_manager.cpp(229) ] Found 1 devices on platform NVIDIA CUDA
   [platform][1688279118][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\opencl\device_manager.cpp(234) ] Found device NVIDIA GeForce GTX 1050 Ti with Max-Q Design on platform NVIDIA CUDA
   [platform][1688279118][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\opencl\device_manager.cpp(229) ] Found 1 devices on platform Intel(R) OpenCL HD Graphics
   [platform][1688279118][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\opencl\device_manager.cpp(234) ] Found device Intel(R) UHD Graphics 630 on platform Intel(R) OpenCL HD Graphics
   [platform][1688279118][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\opencl\device_manager.cpp(229) ] Found 1 devices on platform Intel(R) OpenCL
   [platform][1688279118][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\opencl\device_manager.cpp(234) ] Found device Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz on platform Intel(R) OpenCL
   [platform][1688279118][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\opencl\device_manager.cpp(229) ] Found 1 devices on platform Intel(R) FPGA Emulation Platform for OpenCL(TM)
   [platform][1688279118][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\opencl\device_manager.cpp(234) ] Found device Intel(R) FPGA Emulation Device on platform Intel(R) FPGA Emulation Platform for OpenCL(TM)
   [platform][1688279118][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\opencl\device_manager.cpp(229) ] Found 0 devices on platform Intel(R) FPGA SDK for OpenCL(TM)
   [platform][1688279118][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\opencl\device_manager.cpp(239) ] Found 4 OpenCL devices

3. (optional) Output from the af::info() function if applicable.
   2023-07-01 22:53:14.745 info : Setting CUDA Backend
   2023-07-01 22:53:14.745 info : Testing Backend:
   2023-07-01 22:53:14.745 info : ArrayFire v3.8.3 (CUDA, 64-bit Windows, build 987d5675a)
   Platform: CUDA Runtime 12.0, Driver: 12020
   [0] NVIDIA GeForce GTX 1050 Ti with Max-Q Design, 4096 MB, CUDA Compute 6.1

4. Output from the following scripts:

Run one of the following commands based on your OS

Windows:
Download clinfo from https://github.com/Oblomov/clinfo

If you have NVIDIA GPUs. Run nvidia-smi usually located in
C:\Program Files\NVIDIA Corporation\NVSMI

Provide driver version for your GPU. (This is vendor specific)
536.40

Checklist

• Using the latest available ArrayFire release
• GPU drivers are up to date