Manually added nodes in default config don't join cluster

Start at _add_node_to_cluster in gridengine/driver.py and inspect the qmod invocation, with qbin.py showing how that command is run. Compare hostgroup handling for manually added nodes with the nodearray configuration and the proposed default_hostgroups settings. Done means a manually added node receives the intended hostgroup and joins without the invalid queue error shown.

When manually adding a node, gridengine doesn't assume any hostgroup information.

2021-07-21 16:05:01,816 WARNING: No hostgroups were found for Node(htc-1, ip-0A010529, Standard_F2s_v2, 51ec3a08-c6ee-4af9-a25e-0855a54f2e36). Adding to @allhosts only. Please manually add this node to a hostgroup using qconf. If this was a manually started node, add 'gridengine.hostgroups = @preferred-hostgroup' to the nodearray's configuration section so that future nodes will automatically be added.
2021-07-21 16:05:01,867 ERROR: '/sched/sge/sge-2011.11/bin/linux-x64/qmod -e '@ip-0A010529'' failed: Command '['/sched/sge/sge-2011.11/bin/linux-x64/qmod', '-e', '@ip-0A010529']' returned non-zero exit status 1.
2021-07-21 16:05:01,868 WARNING: Could not add Node(htc-1, ip-0A010529, Standard_F2s_v2, 51ec3a08-c6ee-4af9-a25e-0855a54f2e36) to cluster: Command '['/sched/sge/sge-2011.11/bin/linux-x64/qmod', '-e', '@ip-0A010529']' returned non-zero exit status 1.
2021-07-21 16:05:01,878 ERROR: '/sched/sge/sge-2011.11/bin/linux-x64/qmod -d '@ip-0A010529'' failed: Command '['/sched/sge/sge-2011.11/bin/linux-x64/qmod', '-d', '@ip-0A010529']' returned non-zero exit status 1.
2021-07-21 16:05:01,879 ERROR: b'invalid queue "@ip-0A010529"\n'
Traceback (most recent call last):
File "/opt/cycle/gridengine/venv/lib/python3.6/site-packages/gridengine/driver.py", line 378, in _add_node_to_cluster
self.ge_env.qbin.qmod(["-e", "*@" + node.hostname])
File "/opt/cycle/gridengine/venv/lib/python3.6/site-packages/gridengine/qbin.py", line 142, in qmod
return self._call(_QMOD_PATH, args, check)
File "/opt/cycle/gridengine/venv/lib/python3.6/site-packages/gridengine/qbin.py", line 132, in _call
return check_output([bin] + args)
File "/opt/cycle/gridengine/venv/lib/python3.6/site-packages/gridengine/qbin.py", line 82, in check_output
return _check_output(cmd, args, **kwargs).decode()
File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/usr/lib64/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/sched/sge/sge-2011.11/bin/linux-x64/qmod', '-e', '@ip-0A010529']' returned non-zero exit status 1.

Perhaps we should add a autoscale.json configuration so that manually added nodes (which don't have gridengine.hostgroups set can join hostgroups.

"default_hostgroups": [{"select": "node.nodearray": "hpc", "hostgroups": ["@cyclehpc"]},
{"select": "node.nodearray": "htc", "hostgroups": ["@cyclehtc"]}]