[Refactor][Build] Build the core and all plugins into a single binary by default

Work is already underway in linked pull request #9277, and the task spans plugin registration, code generation, server startup, build scripts, Dockerfiles, CI, and documentation. If taking an unclaimed follow-up, first read the relevant unchecked task and its referenced files; finish that item and verify the stated behavior, such as generated registry freshness or single-binary packaging.

What and why to refactor

DevLake currently compiles every plugin with go build -buildmode=plugin into its own .so (backend/scripts/build-plugins.sh), and the server discovers them at startup by walking PLUGIN_DIR (backend/core/runner/loader.go).

The Go plugin mechanism cannot share a linked image. Each .so embeds its own copy of the core, the helpers, and every third-party dependency it needs. At runtime the loader dedupes packages that the host also links, so RSS is not as bad as disk — but on disk (and therefore in the published container image) the same ~40 MB of shared code is duplicated 47 times.

Measured on the currently published images:

Artifact	amd64	arm64
apache/devlake:main_261028_1424_21af7cd	3.06 GB	2.84 GB
apache/devlake:base (same base, no Go binaries)	306 MB	293 MB
Go payload	≈ 2.75 GB	≈ 2.55 GB

Measured locally (backend/bin):

• lake (server, no plugins): 63 MB
• 4 plugins: 201 MB total
• a single plugin .so (CGO on, unstripped): 65–68 MB (jira 65.2 MB, github 67.7 MB, gitlab 66.4 MB)
• 47 plugins × ~65 MB ≈ 3.0 GB, consistent with the published image

Proof of concept for the alternative — a throwaway main that blank-imports all 47 plugins/<name>/impl packages:

Build	Size
all 47 plugins in one binary, CGO_ENABLED=1 -ldflags "-s -w", incl. gitextractor	47 MB
all 47 except gitextractor, -s -w	31 MB
all 47 except gitextractor, unstripped	46 MB
single plugin alone, CGO_ENABLED=0 (shows the shared ~33 MB floor)	33–38 MB

That probe overstates the saving slightly, because a real server binary links the API, the swagger docs and the service layer too. Measured on the actual server package:

Build	Size
go build ./server with the generated registry, -trimpath -ldflags "-s -w"	106.6 MB
the same, -tags devlake_external_plugins (no plugins)	58.4 MB
go build ./server with the registry, unstripped	148.9 MB

So the Go payload goes from ≈ 2.75 GB to ≈ 107 MB, and the image from ≈ 3.06 GB to ≈ 0.42 GB — about a 7× smaller image, with the plugin code itself accounting for 48 MB of the binary. The 306 MB :base stage is now the largest single piece of the image.

Build time also improves: linking 47 plugins costs ~4–5 s each (~3.5 min at the script's parallelism cap of 4), while linking all 47 into one binary takes ~1.6 s with a warm cache. Compilation is shared either way.

Go itself does not offer a better dynamic plugin story, and this is unlikely to change:

• The plugin package docs list the drawbacks and explicitly recommend the alternative: "it may be simpler for that person or component to generate Go source files that blank-import the desired set of plugins and then compile a static executable in the usual way."
• -buildmode=plugin requires external (cgo) linking, so it forces CGO_ENABLED=1 for the whole server build.
• Host and plugins must match exactly in toolchain version, build tags, flags and dependency versions, otherwise the process dies at load time. Nothing in the repo enforces this.
• Plugins cannot be unloaded, are poorly supported by the race detector, and are unsupported on Windows.
• Go 1.27.0/1.27.1 shipped a runtime regression where loading multiple plugins corrupted stdlib types (golang/go#81303). It is fixed in a patch release, but it is exactly the failure mode this design keeps exposing us to.

So the proposal is: ship the framework, the API and every official plugin as one binary, and keep -buildmode=plugin as the supported way for users to sideload their own plugins or override an official one.

Describe the solution you'd like

Compile-time registration (the Caddy / Grafana / Prometheus pattern). The public plugin interfaces (PluginMeta, PluginTask, PluginModel, …) do not change.

1. Registration helper

Added to core/plugin (shipped):

// MustRegister registers p under the name it reports.
func MustRegister(p PluginMeta) {
    if p == nil {
        panic("plugin: MustRegister called with a nil plugin")
    }
    name := p.Name()
    if name == "" {
        panic("plugin: MustRegister called with an empty plugin name")
    }
    if err := RegisterPlugin(name, p); err != nil {
        panic(err)
    }
}

RegisterPlugin keeps its overwrite semantics, which is what lets a sideloaded .so replace a compiled-in plugin with the same name (§6).

2. Where registration lives

In the generated registry, not in each plugin. No file under plugins/<name>/ had to change: plugins/<name>/<name>.go stays package main, so -buildmode=plugin and standalone go run ./plugins/<name> keep working exactly as before, and the impl packages stay free of registration side effects.

3. Generated registry package

New backend/plugins/all/ (shipped):

//go:build !devlake_external_plugins

// Code generated by tools/pluginsgen. DO NOT EDIT.

package all

import (
    "github.com/apache/devlake/core/plugin"

    gitlab "github.com/apache/devlake/plugins/gitlab/impl"
    // ... one aliased import per plugin
)

func init() {
    plugin.MustRegister(gitlab.Gitlab{})
    // ... one call per plugin
}

tools/pluginsgen locates the plugin.PluginMeta implementer in each plugins/*/impl by parsing the package with go/ast. It requires all three methods (Name, Description, RootPkgPath), so an unrelated Name method cannot be picked by accident, and it fails when a package has none or several.

• DEVLAKE_PLUGINS=a,b filters the registry and none emits an empty one, matching the values scripts/build-plugins.sh already accepts.
• CI regenerates the registry and fails when the checked-in file drifts, next to the existing go mod tidy check.
• plugins/all/external.go carries the devlake_external_plugins tag for the core-only build (§6).

4. Link the registry into the server

server/main.go blank-imports plugins/all. Because the import is unconditional (unless the build tag is set), the linker cannot drop the plugin packages, and plugin.AllPlugins() is populated before runner.LoadPlugins runs.

5. Enforce the dependency direction

The single-binary design makes the layering rule load-bearing: plugins depend on core/helpers, never the reverse. One framework→plugin import would drag that plugin and its whole dependency tree into the core payload and defeat subset builds.

Nothing enforced this, and two test files had already crossed the line:

• helpers/e2ehelper/data_flow_tester_test.go imported plugins/gitlab/{models,tasks}
• server/services/blueprint_makeplan_v200_test.go imported plugins/org/tasks

backend/core/layering_test.go reads the import graph of core/..., helpers/... and server/... — including _test.go files — with go list, and fails on any plugins/ edge. It runs as part of make unit-test, because scripts/unit-test-go.sh already includes the core package.

The only tolerated edge is the composition root: server (i.e. server/main.go) may import the generated plugins/all registry, never an individual plugin. That exception is one explicit map entry, so any new one has to be argued for in review. If we later move the entry point to backend/cmd/lake, the exception disappears entirely.

The two offending tests are made plugin-agnostic rather than allowlisted: the e2ehelper example now uses local fixtures, and the blueprint test builds its mocked org option locally (the plugin is mocked, so the value only has to round-trip).

Two notes for reviewers:

• It is a Go test rather than a depguard rule because .golangci.yaml sets run.tests: false, so lint would not have seen the two violations that actually existed.
• The test opens every framework source file so the Go test cache invalidates when an import is added elsewhere. Without that, a plain go test ./core would replay a cached PASS.

6. Sideloading is a first-class path

The shipping model is "official plugins compiled in, extensions sideloaded":

Mode	Build	Runtime
default	go build ./server → one bin/lake	every official plugin built in, plus any .so found in PLUGIN_DIR
core-only	-tags devlake_external_plugins	framework and API only, every plugin comes from PLUGIN_DIR

LoadGoPlugins was hardened for third-party code (shipped):

• a missing PLUGIN_DIR means "nothing to sideload", not a boot failure;
• a .so that fails to open, has no PluginEntry, or does not implement PluginMeta is logged at error level and skipped, so one bad file cannot stop the server from starting;
• a .so whose name collides with a compiled-in plugin overrides it, with a warning. That is what makes patching an official plugin possible, but it has to be visible.

7. The sideload contract

Sideloading only works if the plugin is built against the same code and toolchain as the binary loading it. Go hashes each shared package and refuses to load a plugin whose copy of, say, core/plugin differs from the host's, failing with plugin was built with a different version of package ....

That makes the following part of the feature rather than documentation polish:

• Publish the build recipe. A .so must be built in the matching builder image (apache/devlake:lake-builder-vX, already published) against the same DevLake tag. A page showing the exact go build -buildmode=plugin invocation is required, otherwise "sideload a plugin" is a trap.
• Version handshake. Add const APIVersion to core/plugin and have every plugin entry package export var PluginAPIVersion = plugin.APIVersion, checked before the type assertion. That turns an opaque package-hash error into "plugin built for API 2, this server speaks API 1". Each of the 47 entry packages gains one line.
• Treat the ABI as public. Every exported type in core/plugin, core/context, core/dal and the domain models is part of the contract once third parties compile against it. Worth stating explicitly so we stop treating them as internal.

8. Build and packaging

• make build → go build -trimpath -ldflags "-s -w" -o bin/lake ./server; make build-plugin stays for core-only and sideload builds.
• backend/Dockerfile: replace the build-plugin + build-server steps with a single build, stop copying /app/bin/plugins, and create an empty PLUGIN_DIR so a volume can be mounted over it.
• backend/Dockerfile.local: same.
• make dev / make godev: run go run ./server after make plugins-registry; no plugin build, no PLUGIN_DIR needed.
• Docs: update the plugin layout section of AGENTS.md, the developer instructions in README.md, and the PLUGIN_DIR note in env.example.

Compatibility

• Plugin authors keep the same interfaces, entry package and directory layout. Nothing under plugins/<name>/ changed; the registry is generated.
• -buildmode=plugin remains fully supported, both for users sideloading extensions and for the core-only build.
• The e2e harness already uses exactly this registration path: backend/test/helper/client.go imports dora, org, refdiff and calls plugin.RegisterPlugin(p.Name(), p) directly. The framework side is therefore already proven today.
• RootPkgPath() — currently documented as "PkgPath information lost when compiled as plugin(.so)" — becomes accurate for static builds. Cleaning that up is deliberately out of scope here.

Task list

• core/plugin: add MustRegister
• tools/pluginsgen + generated plugins/all package (+ external.go build-tag stub)
• server/main.go: link the registry
• core/runner/loader.go: tolerate a missing PLUGIN_DIR, skip broken .so files, warn on overrides
• CI: check the generated registry is up to date
• Dependency-direction guard: backend/core/layering_test.go
• scripts/: unified build; keep build-plugins.sh for core-only and sideload builds
• Makefile (root + backend): build, dev, godev
• backend/Dockerfile and backend/Dockerfile.local: single binary, empty PLUGIN_DIR for sideloaded plugins
• Sideload ABI: core/plugin.APIVersion + PluginAPIVersion export in each plugin entry package
• Docs: how to build and sideload a plugin against a released image; AGENTS.md, README.md, env.example
• CI: assert the release image size budget

Risks and open questions

1. Startup cost — the init() functions of every compiled-in plugin (including the subtask dependency sorter validation in each impl) now run at process start rather than when a shared object is opened. The same work happens today on plugin load, so the total is unchanged; it is now earlier and serialized, and worth measuring.
2. Swagger generation — scripts/swag.sh uses --parseDependency --parseInternal over ./server and ./plugins/*. Adding a registry package should not change the output, but make swag must be re-run and diffed.
3. Migration ordering — registerPluginsMigrationScripts() iterates the plugin map, and registration order changes from "filesystem walk order" to "generated file order". Migration scripts are plugin-namespaced, so this should be safe, but it needs verifying.
4. Silent sideload failures — skipping a broken .so keeps the server up, at the cost of a plugin that just is not there. The error log is the only signal. Consider a PLUGIN_LOAD_STRICT switch that fails startup instead, for operators who need the plugin set to be exact.
5. Subset builds — DEVLAKE_PLUGINS drives build-plugins.sh, swag.sh and now the registry generator. They must agree; a subset that only reaches one of them produces an image that does not match its build arguments.

Related issues

• #9092 — [Proposal] Deprecate Python plugin framework and consolidate on Go. Orthogonal, but both change how plugins are packaged and should be sequenced against each other.

Additional context

Progress lives on feat/single-binary:

• #9277 (merged) — dependency-direction guard.
• #9288 (open) — MustRegister, tools/pluginsgen, the generated plugins/all registry, the registry linked into server, and the hardened sideload path in core/runner/loader.go.
• #9291 (open, stacked on #9288) — make build links one binary, and the image copies only that binary: the .so files never leave the builder stage.

Raw measurements, if anyone wants to reproduce them:

# published image sizes
$ curl -s https://hub.docker.com/v2/repositories/apache/devlake/tags/main_261028_1424_21af7cd | jq '.full_size'
3057398785
$ curl -s https://hub.docker.com/v2/repositories/apache/devlake/tags/base | jq '.full_size'
306008557

# one plugin as a Go plugin (CGO on, unstripped)
$ CGO_ENABLED=1 go build -buildmode=plugin -o /tmp/jira.so ./backend/plugins/jira
$ stat -c %s /tmp/jira.so
65179384

# all 47 plugins in one binary
$ CGO_ENABLED=1 go build -ldflags "-s -w" -o /tmp/all ./probe_all
$ stat -c %s /tmp/all
46976352

The 47 buildable plugins are exactly the directories picked by the current filter in build-plugins.sh (plugins/* minus core, helper, logs, schema_e2e), and all of them have an impl package.