[mypyc] Design discussion: build-system-neutral C generation CLI (mypyc cgen) for projects not using setuptools

Commencez par le point d’entrée existant python -m mypyc et les chemins mypycify()/mypyc_build décrits dans l’issue. Résolvez les questions ouvertes concernant l’emplacement de la CLI, les bases des chemins des schémas, les options de profil du compilateur et sa relation avec mypycify ; le travail est terminé lorsque la conception est approuvée et que la commande proposée ainsi que le contrat JSON versionné sont implémentés, avec une couverture de leur comportement documenté.

Motivation

mypycify() is currently the only supported entry point for compiling Python to C with mypyc, and it is tightly coupled to setuptools/distutils: it returns setuptools.Extension objects, patches build_ext, and requires setuptools to be importable (on Python 3.12+ distutils itself only exists through setuptools).

Projects that build their C extensions with meson, CMake, Bazel, scikit-build-core, or plain Makefiles cannot use mypyc without either dragging setuptools in as a secondary build backend or reverse-engineering mypyc's implicit conventions (shim generation, runtime file copying, include dirs, compiler flags, shared-lib naming, output paths).

This issue proposes a build-system-neutral CLI that performs mypyc's frontend + C generation and emits a machine-readable description of the extensions to build. The actual C compilation stays with the caller's build system.

Proposed CLI

mypyc cgen [--target-dir DIR] [--output-file FILE] [--multi-file] [--separate] SOURCE...

Behavior:

1. Run mypyc's frontend and C generation (type check + codegen, same as mypyc_build).
2. Write all generated C files into --target-dir (default build/mypyc): per-group .c/.h files, the runtime library (lib-rt), and per-module shims when a shared library is used.
3. Emit a JSON "build info" document to --output-file, or to stdout if not given.

Flags:

Deliberately not CLI flags: optimization/debug levels. The emitted cflags contain only flags required for the generated code to work correctly (warning suppression, feature macros like -DMYPYC_LOG_TRACE, the multi-file /GL- workaround on msvc); they never include optimization or debug levels. The consumer's build system owns those decisions, so we don't duplicate the knob at the CLI boundary. (mypycify keeps passing its own defaults, -O3 -g1, so the setuptools path is unchanged.)

JSON build info (schema v1)

{
  "schema_version": 1,
  "target_dir": "/abs/path/to/build/mypyc",
  "extensions": [
    {
      "module": "pkg.mod__mypyc",
      "out_path": "pkg/mod__mypyc.cpython-314-x86_64-linux-gnu.so",
      "sources": ["pkg/__native_pkg.mod.c", "init.c"],
      "include_dirs": ["/abs/.../mypyc/lib-rt", "/abs/path/to/build/mypyc"],
      "depends": ["pkg/__native_pkg.mod.h"],
      "cflags": ["-Werror", "-Wno-unused-function", "..."],
      "link_args": ["-shared"]
    }
  ]
}

Field semantics:

• schema_version: integer, bumped on incompatible changes.
• target_dir: absolute path all relative paths below are resolved against.
• extensions: one entry per extension module to build.
  • module: full dotted extension module name. For a package __init__ shim this follows the setuptools convention "pkg.__init__".
  • out_path: where the built file must end up, relative to the installation root (the equivalent of site-packages), including the platform extension suffix. Note this is a different base than sources/depends (which are relative to target_dir) — this distinction must be explicit in the schema since downstream tooling will join paths.
  • sources: .c files to compile, relative to target_dir (keeps the JSON portable across machines/containers).
  • include_dirs: absolute paths (they point into the mypyc installation, e.g. lib-rt, which cannot be made relative).
  • depends: headers/source files that must trigger a rebuild when they change, relative to target_dir (already transitively resolved from #include scanning, including cross-group export-table headers).
  • cflags: compiler flags required by the generated code (warnings, feature macros). Never includes optimization or debug levels — those are up to the caller's build system.
  • link_args: platform link flags needed to produce a loadable extension module, derived from Python's configured LDSHARED. May be empty; the caller is always responsible for the platform's standard extension linking (e.g. the Python import library on Windows) and for the Python.h include directory.

How a build system would consume this (meson example)

• run_command() or a custom_target() invokes mypyc cgen --output-file ... once per configuration;
• configure step parses the JSON and feeds each entry into py.extension_module(...) using sources, include_dirs, cflags, link_args;
• depends maps to meson's dependency tracking for incremental rebuilds.

A concrete working example (compiling src/bgm_tv_wiki/ast.py; the shim keeps the importable name, the shared library holds the generated code):

project('bgm-tv-wiki', 'c',
  default_options: ['c_std=c11', 'buildtype=release'],
  meson_version: '>= 1.5.0')

py = import('python').find_installation(pure: false)

py.install_sources(
  'src/bgm_tv_wiki/__init__.py',
  'src/bgm_tv_wiki/py.typed',
  subdir: 'bgm_tv_wiki',
)

# ---- 1. Run cgen at configure time: generate C sources + info.json ----
cgen_dir = meson.current_build_dir() / 'mypyc-cgen'

run_command(py, '-m', 'mypyc', 'cgen',
  '--target-dir', cgen_dir,
  '--output-file', cgen_dir / 'info.json',
  meson.project_source_root() / 'src' / 'bgm_tv_wiki' / 'ast.py',
  check: true)

# ---- 2. Extract build parameters from info.json (meson can't parse JSON) ----
json_get = '''
import json, sys
info = json.load(open(sys.argv[1]))
ext = next(e for e in info["extensions"] if e["module"] == sys.argv[2])
print("\\n".join(ext[sys.argv[3]]))
'''

cgen_info = cgen_dir / 'info.json'
native = 'bgm_tv_wiki.ast__mypyc'

native_sources = run_command(py, '-c', json_get, cgen_info, native, 'sources',
  check: true).stdout().strip().split('\n')
native_include_dirs = run_command(py, '-c', json_get, cgen_info, native, 'include_dirs',
  check: true).stdout().strip().split('\n')
native_cflags = run_command(py, '-c', json_get, cgen_info, native, 'cflags',
  check: true).stdout().strip().split('\n')
native_link = run_command(py, '-c', json_get, cgen_info, native, 'link_args',
  check: true).stdout().strip().split('\n')

# sources are relative to target_dir; make them absolute
native_sources = [cgen_dir / s for s in native_sources]
# include_dirs are absolute paths (into the mypyc installation); turn them into -I flags
foreach d : native_include_dirs
  native_cflags += ['-I' + d]
endforeach

# ---- 3. Build the extensions ----
# native: compile the mypyc-generated C (include_dirs already covers lib-rt
# and the cgen output directory). No py.dependency() needed — meson's python
# module provides Python.h and the import library automatically.
py.extension_module(
  'ast__mypyc',
  native_sources,
  subdir: 'bgm_tv_wiki',
  install: true,
  c_args: native_cflags,
  link_args: native_link,
)

# shim: forwards to the native module; only needs Python.h
shim_sources = run_command(py, '-c', json_get, cgen_info, 'bgm_tv_wiki.ast', 'sources',
  check: true).stdout().strip().split('\n')
py.extension_module(
  'ast',
  [cgen_dir / s for s in shim_sources],
  subdir: 'bgm_tv_wiki',
  install: true,
)

Why a CLI (not a Python API)

• Process isolation: the caller's build system can invoke it as a tool without importing mypyc into its own process, and without setuptools installed at all (the CLI path never touches distutils/setuptools).
• The JSON document is a stable, versioned contract between mypyc and any build backend; a Python API would tie callers to mypyc internals and Python packaging conventions.

Open questions

1. Naming/placement: mypyc cgen as a subcommand of the existing python -m mypyc entry point, or a separate console script?
2. Is the schema stable enough to commit to, particularly the "two path bases" convention (sources/depends vs target_dir vs out_path)?
3. Should there be an escape hatch for compiler-profile knobs (e.g. --cflags passthrough), or is "post-edit the JSON" sufficient?
4. Long-term relationship with mypycify: keep both (setuptools users keep mypycify; everyone else uses cgen), or eventually re-implement mypycify on top of cgen?

Happy to work on the implementation once the design is agreed upon.