cmd/compile: Support `make jobserver` for dynamic negotiable concurreny during compilation

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

Not tested

What operating system and processor architecture are you using (go env)?

What did you do?

• Clone five independent packages
• Compile each with say 2 threads
• Observer number of threads
• It goes down from 10 (max possible) to 5 (minimum)

What did you expect to see?

• Using a single workstation as CI server allows orchestration of concurrency during compilation of multiple independent packages
• Compiling go with other languages plays well in regards to optimal utilization of CPU cores

What did you see instead?

• Controlling max threads in a single go compile process is possible
• Sharing of threads between independent go compile process is not possible
• Ensuring that 5 or so independent compile processes are keeping 8 CPU's fully stocked is not feasible.

Possible solution

Implement support for GNU make jobserver for compilation. This doesn't have to be exposed for 80-90% of users whose primary use-case doesn't involve parallel independent compilation of go packages.

Bonus: Both compilation and testing can benefit from this

Pro:

• Works with make, ninja-cmake and other related build and meta build systems
• Works cross-platform on systems with sockets (all modern systems targeted by go as compilation platforms based on expectation of pulling dependencies automatically)
• Parity with compilation process of compilers like C/C++ (clang, gcc), rust (cargo)
• Allows power users to perform quick context switches during compilation and testing
• Allows people working on polyglot code bases a significant boost (a lot of tech stacks are not pure go)

Con:

• jobserver has drawbacks (the pipe could have been closed, env variables need to be passed into sub-shells (not a concern here, but often a source of errors))
• more complicated process for spawning sub-processes
• maybe low utilization in go community