RFC: add `materialize` to materialize lazy arrays
@rgommers is already working on this.
Since Jan 20, 2025.
Assessment
This issue has not been assessed yet.
Description
Preface
I do not think that I am the best person to champion this effort, as I am far from the most informed person here on Lazy arrays. I'm probably missing important things, but I would like to start this discussion as I think that it is an important topic.
The problem
The problem of mixing computation requiring data-dependent properties with lazy execution is discussed in detail elsewhere:
- https://data-apis.org/array-api/draft/design_topics/lazy_eager.html
- https://data-apis.org/array-api/draft/design_topics/data_dependent_output_shapes.html#data-dependent-output-shapes
- https://github.com/data-apis/array-api/issues/748
- https://github.com/data-apis/array-api/issues/834
A possible solution
Add the function materialize(x: Array) to the top level of the API. Behaviour:
- for eagerly-executed arrays, this would be a no-op
- for lazy arrays, this would force computation such that the data is available in the returned array (which is of the same array type?)
- for "100% lazy" arrays (https://github.com/data-apis/array-api/issues/748#issuecomment-2189724261), this would raise an exception
Prior art
- Dask:
- https://docs.dask.org/en/stable/generated/dask.array.Array.compute_chunk_sizes.html computes chunk sizes / shape, working in-place and leaving the array as a Dask array.
- https://docs.dask.org/en/stable/generated/dask.array.Array.compute.html materialises the in-memory equivalent of the dask array, returning e.g. a NumPy array.
- JAX:
- ?
- others?
Concerns
- I think the main concern is whether eager-only libraries will agree to adding a no-op into the API. There is precedent for that type of change (e.g.
devicekwargs in NumPy), but perhaps this is too obtrusive? - As far as I can tell there isn't a standard way to do this across lazy libraries. Does JAX just do this automatically when it would be needed? Do other libraries have this capability?
Alternatives
- Do nothing. The easy option, but it leaves us unable to support lazy arrays when data-dependent properties are used in computation (maybe that is okay?)
- An alternative API. Maybe spelled like
compute*or a method on the array object. Maybe with options for partial materialization (if that's a thing)?
cc @tomnicholas @hameerabbasi @rgommers
- Dominant language
- Python
- Stars
- 281
- Forks
- 52
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from data-apis/array-api
-
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
Maintenance
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
All issues in data-apis/array-api
Similar issues
-
documentation help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
simonw/sqlite-utils#872 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100