Allocation of CUDA devices not retrieved correctly
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 78/100
Research direction
Start in executors.py at the run_job function around line 418, where the issue identifies the CUDA resource lookup. Check the existing resource-allocation behavior for scattered jobs with --parallel and the --debug message for pending jobs. Done means jobs wait for sufficient unallocated CUDA devices and debug output reports the allocated CUDA count correctly.
Written by the indexing model from the issue text.
Description
Expected behavior
When running a workflow where a step uses the "cwltool:CUDARequirement" requirement, the run should be allocating the required number of CUDA devices to each job. When using scattering with "--parallel", new jobs for that step should only be spawned if a sufficient number of unallocated CUDA devices is available.
Current behavior
New jobs are spawned irrespective of the current number of allocated CUDA devices. This leads to multiple jobs being spawned with a single CUDA device and failing jobs due to insufficient resources.
Using "--debug", pending jobs, e.g. due to insufficient RAM, always return "allocated CUDA is 0" even though CUDA devices should be allocated (debug output like 'Job "XXX" cannot be run, requests more resources (XXX) than available on this host (already allocated ram is XXX, allocated cores is XXX, allocated CUDA is 0, max ram XXX, max cores XXX, max CUDA XXX).')
Possible solutions
The Problem was fixed on my machine by editing line 418 in executors.py (run_job function):
cuda = cast(int, job.builder.resources.get("cudaDevices", 0))
with
cuda = cast(int, job.builder.resources.get("cudaDeviceCount", 0))
as "cudaDevices" does not seem to be the correct name of the resource.
- Dominant language
- Python
- Stars
- 376
- Forks
- 256
- Avg merge
- 8h 33m
- Merged PRs (30d)
- 9
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from common-workflow-language/cwltool
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
common-workflow-language/cwltool#2283 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
common-workflow-language/cwltool#2353 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 65/100
common-workflow-language/cwltool#2343 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
common-workflow-language/cwltool#2337 · 2 comments ·
-
`--single-step <stepname>` plus `--help` to list the inputs doesn't produce the correct results Open
Difficulty 3/5 1-2 days Newbie friendliness 58/100
common-workflow-language/cwltool#2299 ·
All issues in common-workflow-language/cwltool
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100