Reductions do not properly support unstructured distributions
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 42/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- python
- Domain
- distributed-systems
Research direction
Run the provided IPython.parallel reproduction with dacluster start -n4, then trace distarray/globalapi/distarray.py through _reduce and distarray/globalapi/maps.py through Distribution.reduce, get_dim_data_per_rank, and UnstructuredMap.get_dimdicts. Done means darr.sum(axis=1) succeeds and the input unstructured indices are propagated to the output distribution.
Written by the indexing model from the issue text.
Description
This reproduces the problem
# Using IPython.parallel: 'dacluster start -n4' before running.
import numpy as np
from distarray.globalapi import Context
from distarray.globalapi.maps import Distribution, NoDistMap, UnstructuredMap
nparr = np.random.rand(4, 5)
context = Context()
m0 = UnstructuredMap(4, 3, [[0], [1], [2, 3]])
m1 = NoDistMap(5, 1)
distribution = Distribution.from_maps(
context,
maps=[m0, m1],
targets=[0, 1, 3]
)
darr = context.fromarray(nparr, distribution)
from IPython.core.debugger import Tracer; Tracer()()
darr.sum(axis=1) # boom!
This results in
Traceback (most recent call last):
File "failure.py", line 19, in <module>
darr.sum(axis=1) # boom!
File "/Users/cfarrow/projects/distarray/distarray/globalapi/distarray.py", line 383, in sum
return self._reduce('sum_reducer', axis, dtype, out)
File "/Users/cfarrow/projects/distarray/distarray/globalapi/distarray.py", line 362, in _reduce
ddpr = out_dist.get_dim_data_per_rank()
File "/Users/cfarrow/projects/distarray/distarray/globalapi/maps.py", line 830, in get_dim_data_per_rank
dds = [enumerate(m.get_dimdicts()) for m in self.maps]
File "/Users/cfarrow/projects/distarray/distarray/globalapi/maps.py", line 510, in get_dimdicts
raise ValueError()
ValueError
Distribution.reduce does not appear to consider UnstructuredMap, which leads to the indices used by the input UnstructuredMap to not be propagated to the output UnstructuredMap.
- Dominant language
- Python
- Stars
- 5
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from enthought/distarray
-
Difficulty 2/5 1-3 hours Newbie friendliness 35/100
-
bug
Difficulty 3/5 1-2 days Newbie friendliness 30/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 35/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 45/100
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 20/100
All issues in enthought/distarray
Similar issues
-
essnmx good first issue
Difficulty 1/5 Under an hour Newbie friendliness 95/100
-
[Feature] 奇物选择添加优先级 Open
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
syfoud/Simulated_Scepter#174 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Giskard-AI/giskard-oss#2840 · 1 comment ·
-
A claim comment carrying the issue number is silently declined while the workflow reports success Openarea: repo bug perceived difficulty: 2
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
yeti-platform/yeti#1380 ·