Proposal: Native ReScript macros for extensions and derivers

Bắt đầu với tài liệu thiết kế PoC và so sánh layout macros/ của riêng dự án cùng các trường trong rescript.json với hạ tầng PPX và extension hiện tại của compiler. Theo dõi luồng request/response Node.js được đề xuất và AST có cấu trúc xuyên suốt quá trình parsing và biên dịch. Ở đây, việc hoàn thành không được xác định bởi một patch nhỏ; trước tiên issue cần có phạm vi ban đầu và kế hoạch triển khai được thống nhất.

Update 26-04-10: Added the sequence diagram for extension and deriver.

Summary

I'd like to propose a native macro system for ReScript.

The goal is to provide a ReScript-native replacement path for custom extension and deriver workflows that currently rely on PPX-style rewriting. Instead of writing transforms in OCaml against the legacy PPX surface, macro authors would write handlers in ReScript and work against a structured public AST.

The proposed user-facing forms are:

• expression extensions: %ns.name(...)
• structure/signature extensions: %%ns.name(...)
• native derivers on type declarations: @ns.name and @ns.name(...)

Motivation

ReScript currently has extensibility through existing PPX-compatible infrastructure, but the authoring model is still inherited from the OCaml world.

That has a few drawbacks for ReScript users:

• custom transforms are not written in ReScript
• the public surface is not designed around ReScript ergonomics
• extension/deriver authoring feels disconnected from the rest of the toolchain
• it is harder to build a ReScript-first ecosystem of compile-time transforms

A native macro system would let us design this around ReScript itself.

Proposal

1. Project-level macro source set

Enable macros from rescript.json:

{
  "name": "@foo/a",
  "sources": {
    "dir": "src",
    "subdirs": true
  },
  "dependencies": ["ppxlib_res"],
  "macros": {
    "dir": "macros",
    "namespace": "foo.a",
    "node": "node"
  }
}

Proposed fields:

• macros.dir
• macros.namespace
• macros.node
  If namespace is omitted, it can default from the package name.

2. Macro registration in ReScript

Macro handlers live under macros/ and are registered with attributes on top-level let bindings:

@macro.extension("graphql")
let expand = (...) => ...

@macro.deriver("spice")
let derive = (...) => ...

3. ReScript-native call-site syntax

Example usage:

let message = %foo.a.graphql("spice")

%%foo.a.graphql("macroValue")

@foo.a.accessors
type user = {name: string}

@foo.a.spice
type drink = {label: string}

4. Structured public AST

Instead of a string-in/string-out protocol, macros should work against a structured public AST API, for example:

• PpxlibRes.Ast
• PpxlibRes.Builder
• PpxlibRes.Payload
• PpxlibRes.Extension
• PpxlibRes.Deriver
• PpxlibRes.Codec

Example extension API:

type ctx = {
  filePath: string,
  moduleName: string,
}

type context =
  | Pexp
  | Pstr
  | Psig

type result =
  | Pexp(Ast.expression)
  | Pstr(array<Ast.structureItem>)
  | Psig(array<Ast.signatureItem>)

Example deriver API:

type ctx = {
  filePath: string,
  moduleName: string,
}

type result = {
  structure: array<Ast.structureItem>,
  signature: array<Ast.signatureItem>,
}

Suggested handler shapes:

@macro.extension("graphql")
let expand = (
  kind: Extension.context,
  payload: Ast.payload,
  ctx: Extension.ctx,
): Extension.result => ...

@macro.deriver("spice")
let derive = (
  ctx: Deriver.ctx,
  args: option<Ast.payload>,
  typeDecls: array<Ast.typeDeclaration>,
): Deriver.result => ...

Intended Direction

This is meant to be the preferred model for new ReScript-authored transforms.

In other words:

• macro authors write ReScript, not OCaml
• the public API is designed for ReScript
• extension points and derivers use ReScript-native syntax
• this reduces the need to introduce new PPX-style rewriting layers for ReScript-specific use cases

Execution Model

A reasonable implementation model is:

1. compile macro sources to JavaScript
2. run them in Node.js during compilation
3. serialize payloads/type declarations across the process boundary
4. expose a structured AST API to macro authors
5. decode the result back into compiler AST and continue compilation
6. The transport format can be JSON, while the public API remains structured AST.

Initial Scope

A first version does not need the full compiler AST.

A useful subset would already cover many cases:

• payloads

  • PStr
  • PSig
  • PTyp
  • PPat

• expressions

  • Pexp_ident
  • Pexp_constant
  • Pexp_apply
  • Pexp_fun
  • Pexp_construct
  • Pexp_let
  • Pexp_sequence
  • Pexp_record
  • Pexp_field

• patterns

  • Ppat_any
  • Ppat_var
  • Ppat_record

• structure/signature items

  • Pstr_eval
  • Pstr_value
  • Pstr_type
  • Psig_value
  • Psig_type

• core types

  • Ptyp_constr
  • Ptyp_arrow
  • Ptyp_tuple

Unsupported nodes can fail explicitly at the macro call site.

Sequence Diagrams

Extension

sequenceDiagram
  participant Src as User source
  participant Parser as Parser
  participant NM as Native_macro
  participant Node as Node runner
  participant Macro as expand

  Src->>Parser: Parse source module
  Parser->>NM: Encounter expression extension
  NM->>NM: Resolve macro registration
  NM->>Node: request JSON
  Node->>Macro: Call expand handler
  Macro-->>Node: Return public AST result
  Node-->>NM: response JSON
  NM->>NM: Decode to compiler AST
  NM-->>Parser: Replace original expression

Deriver

sequenceDiagram
  participant Src as User source
  participant Parser as Parser
  participant NM as Native_macro
  participant Node as Node runner
  participant Macro as registered deriver

  Src->>Parser: Parse source module
  Parser->>NM: Encounter type declaration with deriver
  NM->>NM: Resolve deriver registration
  NM->>NM: Preserve original type and strip macro attrs
  NM->>Node: request JSON
  Node->>Macro: Call registered deriver handler
  Macro-->>Node: Return generated structure and signature items
  Node-->>NM: response JSON
  NM->>NM: Decode to compiler AST
  NM-->>Parser: Append generated items after original type group

Example Outcomes

A simple extension could turn:

let message = %foo.a.graphql("spice")

into something like:

let message = "query:" ++ "spice"

A simple deriver could turn:

@foo.a.spice
type drink = {label: string}

into generated bindings such as:

let encodeDrink: drink => JSON.t
let decodeDrink: JSON.t => drink

while preserving the original type declaration.

PoC

There is already a working reference implementation and design write-up in my fork:

• source code: https://github.com/mununki/rescript
• design document: https://github.com/mununki/rescript/blob/poc-macro/docs/macro_design.md
• PoC example: https://github.com/mununki/rescript-macro-poc

Follow-up Design Questions

The current PoC only covers project-local macros under macros/.

A future design step should define how macros can be published by npm packages and consumed through normal package dependencies. In practice, that means designing the interface for:

• exposing macro entry points from a package
• resolving macro namespaces from dependencies
• using dependency-provided macros from an application project

I think this is an important missing piece for a complete macro design, even though it is intentionally out of scope for the current prototype.