SGE configuration resulting in memory blocked jobs in H-series nodes
@mvrequa is already working on this.
Since Aug 3, 2021.
Assessment
This issue has not been assessed yet.
Description
Cyclecloud version: 8.1.0-1275
OS Image: HPC:CentOS8.1
Problem: Issues with Infiniband connectivity
I was running a small cluster in cyclecloud 7 and recently I updated to cyclecloud 8, the cluster has several node arrays, some non H-series and other H-series for MPI heavy calculations. I noticed that there seems to be a different behaviour in the Infiniband connectivity between 7 and 8. I have tracked part of the problem to the SGE configuration.
The SGE configuration provided in cyclecloud 8.1.0-1275 with the HPC:CentOS8.1 has the variable execd_params set to none. This can result in problems when trying to execute MPI jobs in H-series nodes with Infiniband.
The jobs would fail reporting that the the process cannot allocate the necessary memory, this issue can be manually fixed by setting execd_params H_MEMORYLOCKED=unlimited.
I have tested this using several MPI heavy programs such as VASP and QE , which were running in non H-series nodes without problem, but when trying to run in H-series would consistently fail. After the change of execd_params the programs would run.
However, something I noticed from updating to cyclecloud8 from cyclecloud7 is that running MPI jobs in non H-series jobs using inter node communication seems to no longer be possible.
- Dominant language
- Python
- Stars
- 7
- Forks
- 18
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 7
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Azure/cyclecloud-gridengine
-
Difficulty 3/5 1-2 days Newbie friendliness 42/100
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
-
Difficulty 5/5 Over a week Newbie friendliness 15/100
-
Difficulty 3/5 1-2 days Newbie friendliness 42/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 45/100
All issues in Azure/cyclecloud-gridengine
Similar issues
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
canonical/paas-charm#368 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
tech debt
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
StevenBlack/hosts#3256 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
qualcomm/qai-appbuilder#275 ·