Track DSL module dependencies at compile-time consumption sites
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- elixir
- Domain
- build-system, compilers, tooling
Research direction
Start by reviewing Spark.Dsl.Entity dependency handling and the transformer, verifier, and persister consumption sites described in the proposal. Use the minimal fixture with mix xref trace --label compile to compare Code.ensure_compiled/1 and Macro.compile_apply/4. Done means a defined public helper, an opt-in dependency mode, migrated compile-time consumers, and documented dependency behavior, but the exact API remains open.
Written by the indexing model from the issue text.
Description
- I agree to follow this project's Code of Conduct
- I agree to follow this project's AI Policy, or I agree that AI was not used while creating this issue.
Is your feature request related to a problem? Please describe.
Spark currently determines module dependencies where module-valued DSL fields are declared. Unclassified module references create compile dependencies, while extension authors can opt fields out through no_depend_modules.
This is safe, but it over-approximates dependencies. Whether a module is required at compile time is usually known where a transformer, verifier, or persister consumes it, not where the entity or section field is declared.
As a result, extension authors need to audit all downstream consumers before deciding whether a field belongs in no_depend_modules. The declaration and the code that determines the actual dependency semantics are separated, so they can become inconsistent as an extension evolves.
A module may only be stored as runtime metadata:
Transformer.persist(dsl_state, :implementation, module)
or its metadata may affect validation, transformation, or generated state:
metadata = module.metadata()
Those uses require different dependency semantics even though the module came from the same kind of DSL field.
Structured and polymorphic fields make the declaration-site model even harder to maintain. A field can contain both runtime-only modules and modules whose metadata is consumed during compilation, as discussed in #289.
The conservative default avoids stale compile-time state, but it can also create broad incremental recompilation paths for modules that are never inspected during compilation.
Describe the solution you'd like
I would like Spark to move dependency ownership from entity and section declarations to the transformers, verifiers, and persisters that actually consume module metadata.
The extension-authoring contract would become:
- Module-valued DSL options are dependency-free by default.
- Runtime-only consumers store or invoke the module normally.
- Consumers that only need the module to be available during the current compilation use
Code.ensure_compiled/1. - Consumers whose result affects compile-time validation, transformation, or generated state use a Spark helper backed by
Macro.compile_apply/4. - Spark obtains and supplies the caller's
Macro.Env, so extension authors do not need to interact with compiler internals directly. no_depend_modulescould eventually be deprecated because dependency-free module metadata would become the default.
Conceptually, an entity definition would no longer need to predict how every consumer will use its module fields. The removal of no_depend_modules is intentional:
%Spark.Dsl.Entity{
name: :action,
- no_depend_modules: [:run, :error_handler, :notifiers],
schema: [...]
}
Under the proposed model, module-valued DSL options would not create compile dependencies by default. A transformer, verifier, or persister would register a dependency only when it consumes module metadata during compilation.
A transformer or verifier that consumes module metadata during compilation would declare the dependency at that call site:
metadata =
Spark.Dsl.compile_apply(
dsl_state,
module,
:metadata,
[]
)
A runtime-only consumer would not register a compile dependency:
Transformer.persist(dsl_state, :implementation, module)
If the distinction is already represented in caller-owned DSL state, the consumer can choose the appropriate operation without inspecting the target module:
if entity.compile_time_metadata? do
metadata =
Spark.Dsl.compile_apply(
dsl_state,
entity.implementation,
:metadata,
[]
)
use_metadata(metadata)
else
Transformer.persist(
dsl_state,
:implementation,
entity.implementation
)
end
The exact helper name and namespace are not important. The important change is that dependency semantics are declared where the module is consumed rather than where its field is defined.
Macro.compile_apply/4 is available since Elixir 1.16. Raising Spark's minimum Elixir version to 1.16 does not seem unreasonable for an actively developed framework ecosystem, and it would allow Spark to provide this behavior through a public helper.
This could first be introduced as an opt-in dependency mode and become the default in a future major version after Spark-based extensions have migrated their compile-time module consumption.
Describe alternatives you've considered
Continue maintaining no_depend_modules
This is the current approach and remains the safest default. However, entity authors must understand every transformer and verifier that consumes each field, and the dependency policy can drift away from the implementation.
Support nested dependency policies
Issue #289 proposes controlling dependencies for selected paths inside structured DSL values. This would improve precision, but dependency knowledge would still be encoded separately from the code that consumes the module.
Domain-specific dependency callbacks
For Ash type constraints, a Type-owned callback can identify nested Types whose metadata is required during compilation. This works for that domain, but it is a specialized form of the broader consumption-site dependency model.
Split polymorphic DSL fields
A polymorphic field can be replaced with separate fields whose dependency semantics are explicit in caller-owned DSL state. For example, the run and reactor options discussed in ash-project/ash#2885 would allow ordinary runtime implementations to remain dependency-free while Reactor modules always register a compile dependency.
This is precise for an individual DSL, but it does not provide a general extension-authoring model for Spark.
Additional context
This proposal emerged from a sequence of concrete dependency corrections and reviews rather than from an abstract API preference:
- ash-project/ash#2885 removes compile dependencies for runtime-only DSL modules. Review of the generic action
runfield showed that a polymorphic field can mix runtime-only implementations with Reactor modules whose inputs are consumed by a compile-time verifier. That discussion also led to the possible separation ofrunandreactoroptions. - ash-project/ash#2886 restored compile dependencies for policy check modules because their
init/1results affect compile-time state. - #289 explores nested dependency policies for mixed Ash type constraints. The Type-owned callback suggested there moves dependency knowledge closer to the code that understands and consumes those constraints.
Taken together, these cases suggest that much of the complexity comes from deciding dependency semantics at the declaration site rather than where module metadata is actually consumed.
I created a minimal fixture comparing Code.ensure_compiled/1 and Macro.compile_apply/4:
https://github.com/jechol/elixir-conditional-compile-dependency
The result is:
| Mechanism | Compile ordering | Compile dependency |
|---|---|---|
Code.ensure_compiled/1 |
Yes | No |
Macro.compile_apply/4 |
Yes | Yes |
The fixture verifies the compiler metadata with mix xref trace --label compile and cross-checks the corresponding incremental compilation behavior after changing the target source. It passes with Elixir 1.16.3 / OTP 26 and Elixir 1.20.0 / OTP 29.
There is a trade-off when the dependency decision itself requires inspecting the target module. If a module is currently classified as runtime-only and therefore has no dependency, changing that same module into a compile-time implementation may not make the consumer stale.
That kind of role transition appears uncommon compared with the recurring cost of over-approximated compile dependencies. It may be reasonable to document that changing a module's compile-time role requires a clean compile, while recommending caller-owned discriminators for DSLs where automatic invalidation across role changes is important.
Possible implementation path
If this direction is accepted, a possible implementation path would be:
- Raise Spark's minimum Elixir version to 1.16 so it can use
Macro.compile_apply/4. - Add a common Spark helper that registers a compile dependency using the DSL module's caller
Macro.Env. - Introduce an opt-in mode where module-valued DSL options are dependency-free by default.
- Dogfood the mode in Ash by migrating compile-time module consumption in its transformers, verifiers, and persisters to the new helper.
- Document the extension-authoring contract and provide migration guidance for other Spark-based extensions.
- Compare compile dependency graphs, incremental compilation behavior, and missed invalidation cases in real extensions.
- If the model proves reliable, make consumption-site dependency tracking the default in a future major version.
- Deprecate or remove
no_depend_modulesonly after the default has changed and existing extensions have had a migration path.
The exact helper API, opt-in configuration, and migration timeline remain open for discussion.
- Dominant language
- Elixir
- Stars
- 209
- Forks
- 53
- Avg merge
- 6h 8m
- Merged PRs (30d)
- 9
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Similar issues
-
@swmansion/popcorn: published .d.ts files lack extensions on relative imports, breaking moduleResolution node16/nodenextPossibly taken @Huyvux12 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
software-mansion/popcorn#929 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
ash-project/ash#3127 ·
Maintainers usually reply within 1 day
-
Bug: Table layout and date column are cut off on mobile screens on the Tournaments -> History pageOpenbug
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
hexlet-codebattle/codebattle#2375 ·
-
Bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
wintermeyer/vutuv#2388 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100