Failures when running `taskgraph` generations in parallel

Start with taskcluster/ci/test/kind.yml and run taskgraph full -p taskcluster/test/params in parallel to compare it with sequential runs. Trace the KeyError: 'env' through taskgraph/transforms/job/common.py and the related toolchain entry points named in the log. Done means parallel generations complete without the failure and the workaround is no longer needed.

The VPN repo has a task that ran Taskgraph generations in parallel (via taskgraph full -p taskcluster/test/params). For some reason it has failures when running in parallel, that can't be reproduced when running one at a time (I verified this via interactive task). I also was unable to reproduce the failures when running in parallel on my dev machine.

For now, I've worked around the problem by adjusting the task to run each generation one at a time in a for loop.

Here is a snippet from a failing log:

[task 2023-05-31T20:44:24.167Z] Dumping result with parameters from pull-request-untrusted:
[task 2023-05-31T20:44:24.167Z] See '/builds/worker/.cache/taskgraph/log/vcs' for logs
[task 2023-05-31T20:44:24.167Z] concurrent.futures.process._RemoteTraceback: 
[task 2023-05-31T20:44:24.167Z] """
[task 2023-05-31T20:44:24.167Z] Traceback (most recent call last):
[task 2023-05-31T20:44:24.167Z]   File "/usr/lib/python3.10/concurrent/futures/process.py", line 246, in _process_worker
[task 2023-05-31T20:44:24.167Z]     r = call_item.fn(*call_item.args, **call_item.kwargs)
[task 2023-05-31T20:44:24.167Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/main.py", line 146, in format_taskgraph
[task 2023-05-31T20:44:24.167Z]     tg = getattr(tgg, options["graph_attr"])
[task 2023-05-31T20:44:24.167Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/generator.py", line 171, in full_task_graph
[task 2023-05-31T20:44:24.167Z]     return self._run_until("full_task_graph")
[task 2023-05-31T20:44:24.167Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/generator.py", line 423, in _run_until
[task 2023-05-31T20:44:24.167Z]     k, v = next(self._run)
[task 2023-05-31T20:44:24.167Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/generator.py", line 311, in _run
[task 2023-05-31T20:44:24.167Z]     new_tasks = kind.load_tasks(
[task 2023-05-31T20:44:24.167Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/generator.py", line 77, in load_tasks
[task 2023-05-31T20:44:24.167Z]     tasks = [
[task 2023-05-31T20:44:24.167Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/generator.py", line 77, in <listcomp>
[task 2023-05-31T20:44:24.167Z]     tasks = [
[task 2023-05-31T20:44:24.167Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/task.py", line 1317, in check_run_task_caches
[task 2023-05-31T20:44:24.167Z]     for task in tasks:
[task 2023-05-31T20:44:24.167Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/task.py", line 1246, in check_task_dependencies
[task 2023-05-31T20:44:24.167Z]     for task in tasks:
[task 2023-05-31T20:44:24.167Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/task.py", line 1232, in check_task_identifiers
[task 2023-05-31T20:44:24.167Z]     for task in tasks:
[task 2023-05-31T20:44:24.167Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/task.py", line 1213, in chain_of_trust
[task 2023-05-31T20:44:24.167Z]     for task in tasks:
[task 2023-05-31T20:44:24.167Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/task.py", line 1206, in add_github_checks
[task 2023-05-31T20:44:24.167Z]     for task in tasks:
[task 2023-05-31T20:44:24.167Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/task.py", line 1040, in build_task
[task 2023-05-31T20:44:24.168Z]     for task in tasks:
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/task.py", line 1008, in add_index_routes
[task 2023-05-31T20:44:24.168Z]     for task in tasks:
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/task.py", line 936, in process_treeherder_metadata
[task 2023-05-31T20:44:24.168Z]     for task in tasks:
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/task.py", line 898, in validate
[task 2023-05-31T20:44:24.168Z]     for task in tasks:
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/task.py", line 886, in task_name_from_label
[task 2023-05-31T20:44:24.168Z]     for task in tasks:
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/task.py", line 842, in set_defaults
[task 2023-05-31T20:44:24.168Z]     for task in tasks:
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/task.py", line 821, in set_implementation
[task 2023-05-31T20:44:24.168Z]     for task in tasks:
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/cached_tasks.py", line 64, in cache_task
[task 2023-05-31T20:44:24.168Z]     for task in order_tasks(config, tasks):
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/cached_tasks.py", line 22, in order_tasks
[task 2023-05-31T20:44:24.168Z]     pending = deque(tasks)
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/job/__init__.py", line 365, in make_task_description
[task 2023-05-31T20:44:24.168Z]     configure_taskdesc_for_run(
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/job/__init__.py", line 439, in configure_taskdesc_for_run
[task 2023-05-31T20:44:24.168Z]     func(config, job, taskdesc)
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/job/toolchain.py", line 165, in docker_worker_toolchain
[task 2023-05-31T20:44:24.168Z]     common_toolchain(config, job, taskdesc, is_docker=True)
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/job/toolchain.py", line 102, in common_toolchain
[task 2023-05-31T20:44:24.168Z]     docker_worker_add_artifacts(config, job, taskdesc)
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/job/common.py", line 101, in docker_worker_add_artifacts
[task 2023-05-31T20:44:24.168Z]     taskdesc["worker"]["env"]["UPLOAD_DIR"] = path
[task 2023-05-31T20:44:24.168Z] KeyError: 'env'
[task 2023-05-31T20:44:24.168Z] """
[task 2023-05-31T20:44:24.168Z] 
[task 2023-05-31T20:44:24.168Z] The above exception was the direct cause of the following exception:
[task 2023-05-31T20:44:24.168Z] 
[task 2023-05-31T20:44:24.168Z] KeyError: 'env'