C++ toolchain callbacks

Description

Add the option to create a Starlark method tied to the C++ toolchain callable before actions

There are many feature requests that involve adding code to the Bazel C++ rules in order to pipe the variables necessary for very specific use cases:

• a new output by the compilation or linking actions not yet contemplated by the rules

• flags that have to be generated dynamically and therefore cannot go in copts or linkopts but for which there isn’t an existing build variable

It would scale better if the C++ rules code base didn’t have to add explicit support for each of these use cases.

Work

The community will drive adding support for this feature, it first requires discussion in a design doc and reviewing existing issues labeled team-rules-Cpp in bazelbuild/bazel to make sure that we consider every use case that could benefit from this. Once the design is approved, the author can proceed with implementation.

Issues that could benefit (important to find more to justify the work here):

1. Custom build variable with short filename: https://github.com/bazelbuild/bazel/issues/15924

2. AOSP issue for linking which can benefit from this: https://github.com/bazelbuild/bazel/issues/17277

Possible implementation

(This is an idea of how I imagine it roughly, final implementation should be guided by the design.)

Placement

Currently the C++ rules are structured as follows:

Compilation

1. Rule outer later (cc_library, cc_binary, etc..)

2. CcCompilationHelper.java (we will start rewriting to Starlark this quarter)

3. CppCompileActionBuilder

4. CppCompileAction

Linking

1. Rule outer later (cc_library, cc_binary, etc..)

2. CcLinkingHelper

3. CppLinkActoinBuilder

4. CppLinkAction

The toolchain callback should probably go between 2 and 3, in other words, it should be called once for every CppCompileAction and every CppLinkAction created. This allows adding the custom output per action.

Possible callback signature for CppCompileAction

def toolchain_callback(ctx, source): # is cc_toolchain and feature config needed  here?
        # Logic specific to each implementation goes here
        # …..
	# Output which would then be read from the C++ rule’s implementation and passed
        # when constructing the CppCompile action
	return {
                 “additional_inputs” : new_inputs,
                 “additional_outputs” : new_outputs,            
                 “extensions” : extensions, # Build variable extensions for the crosstool
            }

The extensions would be a simple dictionary like the one passed here.

For https://github.com/bazelbuild/bazel/issues/15924, the flags can be created in a loop one by one based on the source passed.

There are plans for Q1 and Q2 to starlarkify CcCompilationHelper but meanwhile the part of the C++ rules logic that would invoke the callback is still in Java despite the toolchain callback being in Starlark. Calling Starlark code from Java is possible via Starlark.call (see here, here and here).

The compilation callback would be stored in the CcToolchainProvider and called before creating every CppCompilation action. Similarly for the linking callback. The callback can be passed as an argument here. This feature request only makes sense for projects that are using Bazel's C++ rules and want to add customization on top, a project with custom rules doesn't need this feature.

The suggestions above can be modified completely after experimenting with real use cases. After the design discussion and experimenting contributors may also conclude that this isn't really needed and there are better ways to achieve the same thing.