Proposal: Automatic Merging of Per-Platform Packages

Start by reading the existing tool package and its tests, then trace how llcppg’s existing writer consumes go/ast files. Add the proposed PlatformPackage and Merge API in merge.go, following the specified normalization, symbol comparison, file placement, and build-constraint rules. Done means valid inputs merge deterministically into shared and z_$(GOOS)_$(GOARCH).go files, with no input mutations beyond the documented annotation and focused tests covering equality and platform-specific declarations.

Summary

Proposal #1125 makes llcppg generate one independent Go package per $GOOS-$GOARCH platform. That solves generation, but leaves the consumer with N near-identical packages (darwin-arm64, linux-amd64, ...) where, in practice, 90% or more of the declarations are the same on every platform.

This proposal adds a small, self-contained capability, placed in the existing tool package (no new package is created), that merges the per-platform ASTs into one multi-platform Go package:

• Declarations that are identical on every platform are emitted once, in a shared file.
• Declarations that differ on any platform (or exist on only some platforms) are emitted per platform, in z_$(GOOS)_$(GOARCH).go files guarded by build constraints.

The merge works on go/ast values, so it is independent of how the platform packages were produced. It does not touch the filesystem, run clang, or format source code.

Motivation

After #1125, running llcppg for darwin/arm64 and linux/amd64 on libc produces two packages. A consumer has to pick one by hand, and a reviewer has to diff them by eye to see what really differs. Typical libc output looks like this:

• getpid, malloc, strlen and most functions: identical everywhere.
• Mode_t, Stat, O_NONBLOCK, EAGAIN: different (or only present) on some platforms.

What we want is the layout Go programmers already know from the standard library: a common file, plus platform files selected by build constraints. Doing that merge mechanically, and deterministically, is what this proposal specifies.

The comparison has to be smarter than reflect.DeepEqual on two ASTs, because generated ASTs differ in ways that do not matter:

• Positions. Every node carries a token.Pos that differs per file.
• Comments. Doc comments are copied from headers, and headers differ per platform.
• Parameter names and function bodies. The C headers may name a parameter buf on one platform and buffer on another; the Go signature is still the same.
• Package aliases. c.Int in one file and libc.Int in another (different import aliases) refer to the same type.

At the same time, comparison must not ignore directives such as //go:linkname and //llgo:link, because they change what a declaration means at link time.

Scope

Placement: the code lives in the existing tool package as new exported API (PlatformPackage, Merge), in its own source file(s) (for example merge.go). It does not introduce a new package or a new module dependency.

In scope: the input/output model, the position-independent normalization, symbol identity, the equality rules, and file placement.

Out of scope (see Non-Goals): formatting and writing files, choosing which platforms to merge, merging a subset of platforms into one //go:build a || b file, and any change to llcppg's cfg schema.

Input and Output

Each input is one platform-specific package, as produced by llcppg for that platform:

// PlatformPackage is the converted output of llcppg for one platform.
type PlatformPackage struct {
	Tags  []string             // $GOOS = Tags[0], $GOARCH = Tags[1]
	Files map[string]*ast.File // file name => AST
}

The output is the merged package:

// Merge merges per-platform packages into one multi-platform package.
// The result maps file name => AST.
func Merge(pkgs []*PlatformPackage) (map[string]*ast.File, error)

Usage is deliberately small:

merged, err := tool.Merge([]*tool.PlatformPackage{
	{Tags: []string{"darwin", "arm64"}, Files: darwinFiles},
	{Tags: []string{"linux", "amd64"}, Files: linuxFiles},
})
if err != nil {
	return err
}
for name, f := range merged {
	// hand f to whatever writes Go source (for example the existing
	// llcppg writer); name is the output file name, e.g. "stdio.go"
	// or "z_linux_amd64.go".
}

Input requirements, each checked up front and reported as an error:

• len(Tags) == 2, both non-empty, and the pair is unique across inputs.
• All platforms declare the same package name.
• Within one platform, a global symbol is declared only once.
• No input file name starts with z_ (that prefix is reserved for generated platform files).

The input ASTs are treated as read-only. The only change Merge makes to them is the package-ref annotation in Ident.Obj.Data (Step 1), a field that exists to carry extra information. The output ASTs are new trees: the chosen declarations are copied, and every token.Pos in the copies is token.NoPos.

The Algorithm

The merge has six steps. Steps 1 and 2 build the data, steps 3 to 5 decide where each symbol goes and in what order, and step 6 assembles the output files.

Step 1: Make package references independent of the import name

Comparison (Step 5) must not depend on positions or on the local name an import happens to have. Positions need no preparation: the input token.Pos values are not cleared. They are simply never compared (Step 5 ignores them), and Step 3 reads them to compute symbol offsets. Positions are dropped only in the generated output (Step 6, where every Pos is token.NoPos).

What does need preparation is every reference to another package, which is made explicit and independent of the local import name. A SelectorExpr whose X is an Ident naming an imported package (c.Int, unsafe.Pointer, libc.Size_t) is marked as an extern package ref. The marker is stored on the identifier itself:

// importPath is the full import path of a package, for example
// "github.com/goplus/lib/c".
type importPath string

// For an extern package ref  X.Sel  where X is an *ast.Ident:
//   X.Obj  = &ast.Object{Kind: ast.Pkg, Name: X.Name, Data: importPath("github.com/goplus/lib/c")}

(go/ast calls the field Ident.Obj; its Data field is where the import path is stored. If X.Obj is nil it is created.) Writing the marker into the input AST goes against the principle that inputs are not modified, but Obj.Data is a field meant for recording extra information, and the annotation is idempotent, so this is the one deliberate exception.

How an identifier is recognized as a package name:

• Resolve it against the file's own import declarations: explicit alias if present, otherwise the imported package's name. Dot imports and blank imports never produce package refs.
• An identifier that is shadowed by a local declaration (parameter, field, local variable) is not a package ref. In generated code this is rare, but the rule must still be applied so a parameter named c is not mistaken for package c.

From this point on, two extern package refs are equal if and only if their importPath and Sel.Name are equal. The local name X.Name plays no role.

The local names are remembered (importPath => local name) so imports can be regenerated at the end (see Step 6).

Step 2: Collect the global symbols

A global symbol is a top-level name in the package scope. The symbol key is:

Declaration	Key
func	function name
method	RecvType.Method (pointer-ness of the receiver is not part of the key)
type	type name
var	name
const block	first name in the block (the whole block is one symbol)

Rules for collecting:

• A var or type GenDecl with several specs is split into one symbol per spec, so var ( A int; B int ) yields two symbols.
• A const block is always one symbol, as a whole. Every const declaration is a single unit, keyed by the first name in the block, whether or not it uses iota or implicit repetition, and whether or not it is parenthesized (const X = 1 is simply a block of one). The block is placed, compared and emitted as a whole; it is never split across files and never partially shared. This keeps C enum bindings (const ( RED Color = iota; GREEN; BLUE )) intact and makes the rule easy to state: the block is the unit, not its members.
• init functions and blank (_) names are not symbols and cannot be merged. They are an error in input.
• For every symbol we record, per platform, its declaration, the file it came from, and its offset in that file (the Offset defined in Step 3). The offset is computed from the input positions, which are still intact.

Name collisions involving a const block are handled as follows: if some name inside a const block on one platform is a separate symbol (or belongs to a different const block) on another platform, all involved symbols are treated as non-identical (Step 4). This keeps the output free of duplicate declarations.

Step 3: Decide the home file of each symbol

Each symbol S has a home file, the file in which its shared copy would live.

• If all platforms declare S in the same file name, that file is the home.
• If platforms declare it in different file names, the home is the lexicographically smallest file name among them. Call that name $(Fname) (the file name with the .go suffix removed).

Example: Mode_t is in sys_types.go on darwin and in types.go on linux. The home is sys_types.go.

The home file only matters for symbols that end up shared (Step 4). It is also what keeps the merged package looking like the generated ones: a symbol stays in a file whose name came from the real header.

The Offset of a symbol

The same idea, extended to also decide the order of symbols inside a file (Step 6), is the symbol's Offset. On each platform P (written $GOOS-$GOARCH, for example darwin-arm64), a symbol S has:

Offset(S, P) = ($Fname, P, $Foffset)

• $Fname is the name of the file in which P declares S.
• P is the platform string.
• $Foffset is the byte offset of S inside that file on P.

Offsets are compared lexicographically, component by component: file name first, then platform string, then file offset. Offsets from different files or platforms are therefore never compared by $Foffset alone.

$Foffset is the distance from the start of the file to the start of the declaration, computed from the input AST as int(node.Pos() - file.FileStart) (so no token.FileSet is needed). For a split var or type declaration the node is the spec; for a const block, the whole block; for a function, the func declaration. If an input file carries no usable positions, the index of the declaration in File.Decls (and of the spec within it) is used instead, consistently for that whole file.

For a shared symbol (identical on all platforms), the symbol's single Offset is the smallest of its per-platform Offsets:

Offset(S) = min over P of Offset(S, P)  =  ($Fname, P*, $Foffset)

This one value fixes everything about the shared copy: $Fname is the home file (it is exactly the smallest-file-name rule above), $Foffset is the position that orders it, and P* is the platform whose copy of S is used, including its doc comment. P* is the first platform, in sorted order, among those that declare S in the home file.

For a platform-specific symbol, each platform's copy keeps its own Offset(S, P).

Step 4: Shared or per-platform

For each symbol S:

• Identical on all platforms (it exists on every input platform and all copies are equal under the rules in Step 5): emit one copy into $(Fname).go.
• Otherwise (it differs on at least one platform, or is missing on at least one platform): do not share it. Emit each platform's own copy into that platform's file z_$(GOOS)_$(GOARCH).go.

Consequences worth spelling out:

• A symbol that exists on only some platforms is always platform-specific.

• There is no "mostly the same" mode. If it is identical on darwin-arm64 and linux-arm64 but different on linux-amd64, all three get their own copy. Grouping platforms is left for a follow-up.

• Platform files carry an explicit build constraint:

  //go:build darwin && arm64
  

  Because the file name has the form z_$(GOOS)_$(GOARCH).go, with underscores, Go already derives the same constraint implicitly from the _GOOS_GOARCH.go suffix, so the explicit line is redundant. It is still written, so the constraint is visible in the file itself and does not depend on file-name parsing. The z_ prefix makes platform files sort after the shared files.

• With a single input platform every symbol is trivially identical, so everything lands in the shared files and no z_ file and no build constraint is produced.

• A file that would contain no declarations is not emitted.

Step 5: When are two declarations "the same"?

Comparison works on the input ASTs as annotated in Step 1, and follows these rules.

What is ignored

• All comments, with the exception of directives (below).
• Positions.
• For functions and methods: the body, and the names of parameters, results and the receiver. (llcppg-generated functions normally have no body, but the rule does not depend on that.)
• Struct field doc/line comments.

What is always compared

• Directives. A directive is a comment line of the form //word:... with no space after //, for example //go:linkname Getpid C.getpid or //llgo:link (*T).Close C.close. They are collected from the doc comment of the declaration (including a GenDecl doc for single-spec declarations), compared as an ordered list of lines with insignificant whitespace normalized. //go:build lines are not symbol directives and are managed by the merge itself.
• For functions: name, directives, type parameters, and the parameter and result types (including variadic-ness), plus the receiver type for methods.
• For types: the name, whether it is an alias (type A = B) or a definition, type parameters, and the full underlying type expression. Struct fields are compared by name, type, tag and embedding; field order matters.
• For vars: name, declared type, and the value expression.
• For const blocks: the whole block is compared as one unit: the same number of specs, in the same order, with the same names, declared types and value expressions (including implicit repetition and iota). A block that differs in even one member is not identical, so the entire block becomes platform-specific.

How selector expressions are compared

• If both sides are extern package refs (Step 1), compare importPath and Sel.Name.
• If only one side is, they are different.
• Otherwise (ordinary field or method selection) compare X and Sel with the normal structural comparison.

Other identifiers are compared by name only; Ident.Obj for local scopes is ignored.

The comparison is structural and deterministic, and returns the same answer regardless of the order the platforms were given.

Step 6: Assemble the output files

After the placement decisions, the result is assembled:

• Declarations. For a shared symbol, the copy from platform P* is used (see "The Offset of a symbol" in Step 3), which also decides which doc comment survives. Platform-specific symbols keep their own platform's declaration and comments. Within one output file, declarations are sorted by their Offset, in ascending order:

  • In a shared file $(Fname).go, each symbol is sorted by its single Offset(S) = ($Fname, P*, $Foffset). All entries share the same $Fname, so the order is by P*, then $Foffset.
  • In a platform file z_$(GOOS)_$(GOARCH).go, each symbol is sorted by its own Offset(S, P) for that platform, so the order is by source file name, then offset within that file.

  Offsets are unique, so the order is total. It does not depend on the order in which platforms were passed in, and it keeps the logical grouping that llcppg produced from the headers: a type next to its methods, a const block next to the type it belongs to.

  A typical shared file is simply the source order of the first platform: when every platform splits files the same way, every shared symbol has the same P*, and the order is that platform's own order. When platforms split files differently, symbols are grouped by the platform that supplied them (first platform first), and by source offset inside each group. This only affects readability, not correctness.

• Imports. Imports are regenerated per output file from the importPath refs actually used in it, using the remembered local names. If two platforms disagree on the local name for the same path and both end up in one file, the name from the first platform in sorted order is used and every reference in that file follows it, which is safe because references are by import path. A file that contains any //go:linkname directive additionally gets import _ "unsafe" unless it already imports unsafe.

• Package clause. The common package name.

• Positions. The output files are built from copies of the chosen declarations in which every token.Pos is token.NoPos. Input positions are used only for ordering (Step 3) and never leak into the output.

• Build constraints. Added to z_* files as described in Step 4.

Worked Example

Two platforms, three symbols:

darwin-arm64 types.go:

package libc

import "github.com/goplus/lib/c"

type Mode_t uint16

//go:linkname Getpid C.getpid
func Getpid() c.Int

linux-amd64 libc_types.go (note the different alias, a different parameter name elsewhere does not matter either):

package libc

import libc "github.com/goplus/lib/c"

type Mode_t uint32

//go:linkname Getpid C.getpid
func Getpid() libc.Int

Result:

Symbol	Equal everywhere?	Output file
Getpid	yes (c.Int and libc.Int are both github.com/goplus/lib/c.Int)	libc_types.go (smallest of libc_types.go and types.go)
Mode_t	no (uint16 vs uint32)	z_darwin_arm64.go and z_linux_amd64.go

z_darwin_arm64.go starts with //go:build darwin && arm64 and contains type Mode_t uint16; the linux file contains the uint32 version under //go:build linux && amd64.

If Getpid had carried //go:linkname Getpid C._getpid on linux, the directive difference alone would make it platform-specific.

Errors

Merge returns an error, without partial output, for:

• Invalid or duplicate platform tags.
• Package name mismatch between platforms.
• A symbol declared twice on one platform, or an init/blank top-level declaration.
• An input file whose name collides with a generated z_ file name.
• An import that cannot be resolved while normalizing (a selector whose X looks like a package but is not in the file's imports and has no local declaration).

Compatibility

This is a new capability added to the existing tool package: it only adds new exported API (PlatformPackage, Merge) and changes no existing API. No existing cfg field or command-line behavior changes. The merged output is ordinary Go and builds with the standard toolchain on each target.

Non-Goals

• Writing or formatting the output. Merge returns ASTs; since output positions are all token.NoPos, whichever component prints them needs to supply positions (or use a printer that does not depend on them).
• Selecting platforms, running llcppg per platform, or wiring this into the llcppg command line (for example a -merge flag). That is a follow-up on top of #1125.
• Grouping platforms (darwin || linux) to reduce duplication between platform files.
• Merging in the other direction (splitting a package back into platforms).
• Semantic equivalence. Two declarations that are different spellings of the same type (for example int32 vs c.Int that happens to be 32 bits) are different; the comparison is syntactic on purpose, since llcppg's choice of spelling carries meaning.

Open Questions

1. Cgo-style local names. Should the algorithm treat packages the same name under different paths (vendored copies of lib/c) as equal? Proposed answer: no, the import path is the identity.
2. Representative comments. The doc comment of a shared symbol comes from the platform P* that supplies its minimum Offset, which is simple and deterministic; an alternative is to prefer the platform whose comment is longest. Proposed: keep the simple rule.
3. Subset grouping. Whether a follow-up should detect identical copies on a subset of platforms and emit one //go:build expression for them, and how to name such files.
4. Const blocks. Because a const block is atomic, a block that differs in a single member (for example one extra errno value on one platform) is duplicated in full on every platform. Splitting such blocks by member is deliberately not done; if it proves too noisy in practice, it can be revisited in a follow-up.
5. Offset tie-break by platform. Because the platform string is the second component of an Offset, a shared symbol is ordered and sourced from the first platform (in sorted $GOOS-$GOARCH order) that declares it in the home file. Should the caller be able to prefer a particular platform (for example, the one with the most complete headers)? Proposed: not initially, to keep Merge independent of input order.