Can't run pennyLane benchmarks in 23.10 cuQuantum Appliance

The issue was also present in the 23.06 container as well.

Example to reproduce error:

cuquantum-benchmarks circuit --frontend pennylane --backend pennylane-lightning-gpu --benchmark qaoa --nqubits 16

Source of error:
Line 46
https://github.com/NVIDIA/cuQuantum/blob/main/benchmarks/cuquantum_benchmarks/backends/backend_pny.py

Full trace:

Traceback (most recent call last):
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/cuquantum_benchmarks/backends/backend_pny.py", line 46, in find_version
    import pennylane_lightning_gpu
ModuleNotFoundError: No module named 'pennylane_lightning_gpu'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cuquantum/conda/envs/cuquantum-23.10/bin/cuquantum-benchmarks", line 8, in <module>
    sys.exit(run())
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/cuquantum_benchmarks/run.py", line 335, in run
    runner.run()
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/cuquantum_benchmarks/run_interface.py", line 90, in run
    self._run()
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/cuquantum_benchmarks/run_interface.py", line 267, in _run
    backend = createBackend(
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/cuquantum_benchmarks/backends/__init__.py", line 34, in createBackend
    return backends[backend_name](ngpus, ncpu_threads, precision, *args, **kwargs)
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/cuquantum_benchmarks/backends/backend_pny.py", line 40, in __init__
    self.version = self.find_version(identifier) 
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/cuquantum_benchmarks/backends/backend_pny.py", line 48, in find_version
    raise RuntimeError("PennyLane-Lightning-GPU plugin is not installed") from e
RuntimeError: PennyLane-Lightning-GPU plugin is not installed

There is a similar error present when trying to run the kokkos backend as well.

The libraries are present when doing conda list but not when importing in a script (or with interactive Python).

conda list

pennylane                 0.35.1                   pypi_0    pypi
pennylane-lightning       0.35.1                   pypi_0    pypi
pennylane-lightning-gpu   0.35.1                   pypi_0    pypi

Here is the docker file I used for building the image:

FROM  nvcr.io/nvidia/cuquantum-appliance:23.10
RUN git clone https://github.com/NVIDIA/cuQuantum.git \
&& cd cuQuantum/benchmarks \
&& pip install .[all]

(Also had the same error using docker commit route).
I also tried rebuilding cuquantum-benchmarks to fix the problem. This didn’t help.

Suspected solution:

Maybe the import statements are wrong. From a brief look at the pennyLane documentation I found a different method for importing.

import pennylane as qml
dev = qml.device("lightning.gpu", wires=2)

This at least throws me a cuda error:

/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/pennylane_lightning/lightning_gpu/lightning_gpu.py:72: UserWarning: libcudart.so.12: cannot open shared object file: No such file or directory
  warn(str(e), UserWarning)
/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/pennylane_lightning/lightning_gpu/lightning_gpu.py:1014: UserWarning: 
                "Pre-compiled binaries for lightning.gpu are not available. Falling back to "
                "using the Python-based default.qubit implementation. To manually compile from "
                "source, follow the instructions at "
                "https://pennylane-lightning.readthedocs.io/en/latest/installation.html.",
            
  warn(

This is being solved here:
https://discuss.pennylane.ai/t/pennylane-lightning-gpu-0-35-on-cuquantum-appliance-23-10/4393

Related bug:

I also noticed the CPU backend does not work in the 23.10 container either (This ran for me in the 23.06 container):

cuquantum-benchmarks circuit --frontend pennylane --backend pennylane --benchmark qaoa --nqubits 16

024-04-05 13:04:25,345 INFO     * Running qaoa with 1 CPU threads, and 16 qubits [pennylane-v0.35.1 | pennylane-v0.35.1]:
Traceback (most recent call last):
  File "/home/cuquantum/conda/envs/cuquantum-23.10/bin/cuquantum-benchmarks", line 8, in <module>
    sys.exit(run())
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/cuquantum_benchmarks/run.py", line 335, in run
    runner.run()
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/cuquantum_benchmarks/run_interface.py", line 90, in run
    self._run()
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/cuquantum_benchmarks/run_interface.py", line 301, in _run
    preprocess_data = backend.preprocess_circuit(
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/cuquantum_benchmarks/backends/backend_pny.py", line 114, in preprocess_circuit
    self.circuit = self._make_qnode(circuit, nshots, **kwargs)
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/cuquantum_benchmarks/backends/backend_pny.py", line 101, in _make_qnode
    dev = pennylane.device("default.qubit", wires=self.nqubits, shots=nshots, c_dtype=self.dtype)
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/pennylane/__init__.py", line 378, in device
    dev = plugin_device_class(*args, **options)
TypeError: DefaultQubit.__init__() got an unexpected keyword argument 'c_dtype'