Errors with big arrays and many communicators

Open

#556 opened on Aug 5, 2014 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty: 4/5
Estimated time: 3-5 days
Newbie friendliness: 35/100

Issue type: Bug
Clarity: Needs clarification
Activity status: Stale
Tech stack: python
Domain: distributed-systems

Research direction

Start by reproducing the reported commands from benchmark_julia.py and trace execution through distarray/mpi_engine.py, distarray/dist/context.py, distarray/local/maps.py, and distarray/local/construct.py. Compare the mpi4py send/pickle and Create_cart failures to determine their causes; done means the benchmark runs without the reported errors or clearly reports the unsupported condition.

Written by the indexing model from the issue text.

Description

bug

I ran into some errors using the julia_set benchmark:

First, I tried to shoot the moon and see if we could go up to 16384:

$ mpiexec -n 1 python benchmark_julia.py -r6 -olong_run.json 128 256 512 1024 204
8 4096 8192 16384 : -n 12 python -m distarray.apps.engine
...
(343/2352: 0.895s) (1406335077.099205, 1406335077.99378, 'c-c', 8192, '(-0.045+0.45j)', 12, [174749288L, 174751472L, 174748776L, 174752138L, 174748831L, 174749623L, 174751680L,
 174749623L, 174747481L, 174750332L, 174746970L, 174749666L])
Traceback (most recent call last):
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/rgrant/development/distarray/distarray/apps/engine.py", line 9, in <module>
    Engine()
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 41, in __init__
    val = self.parse_msg(msg)
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 80, in parse_msg
    ret = func(msg)
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 118, in func_call
    self._INTERCOMM.send(res, dest=self.client_rank)
  File "Comm.pyx", line 800, in mpi4py.MPI.Comm.send (src/mpi4py.MPI.c:66431)
  File "pickled.pxi", line 180, in mpi4py.MPI.PyMPI_send (src/mpi4py.MPI.c:27319)
  File "pickled.pxi", line 81, in mpi4py.MPI._p_Pickle.dump (src/mpi4py.MPI.c:26188)
SystemError: NULL result without error in PyObject_Call

Notice that it handled all the runs up through 8192x8192

Then, I tried to just do a full run of 6 reps of up to 8192x8192:

$ mpiexec -n 1 python benchmark_julia.py -r6 -olong_run_2.json 128 256 512 1024 2
048 4096 8192 : -n 12 python -m distarray.apps.engine
...
(2042/2058: 1.267s) (1406336596.783966, 1406336598.050912, 'c-c', 8192, '(-0.045+0.45j)', 8, [262124206L, 262123056L, 262123488L, 262123056L, 262121394L, 262128371L, 262123938L, 262128371L])
Traceback (most recent call last):
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 162, in _run_module_as_main
Traceback (most recent call last):
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 162, in _run_module_as_main
Traceback (most recent call last):
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 162, in _run_module_as_main
Traceback (most recent call last):
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
Traceback (most recent call last):
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/rgrant/development/distarray/distarray/apps/engine.py", line 9, in <module>
    Engine()
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 41, in __init__
    val = self.parse_msg(msg)
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 80, in parse_msg
    "__main__", fname, loader, pkg_name)
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/rgrant/development/distarray/distarray/apps/engine.py", line 9, in <module>
    Engine()
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 41, in __init__
    val = self.parse_msg(msg)
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 80, in parse_msg
    ret = func(msg)
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 115, in func_call
    "__main__", fname, loader, pkg_name)
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/rgrant/development/distarray/distarray/apps/engine.py", line 9, in <module>
    Engine()
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 41, in __init__
    val = self.parse_msg(msg)
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 80, in parse_msg
    ret = func(msg)
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 115, in func_call
    res = new_func(*args, **kwargs)
  File "/home/rgrant/development/distarray/distarray/dist/context.py", line 133, in create_local
    distribution = Distribution(comm=comm, dim_data=dim_data)
  File "/home/rgrant/development/distarray/distarray/local/maps.py", line 51, in __init__
Traceback (most recent call last):
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/rgrant/development/distarray/distarray/apps/engine.py", line 9, in <module>
    Engine()
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 41, in __init__
    val = self.parse_msg(msg)
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 80, in parse_msg
    ret = func(msg)
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 115, in func_call
    res = new_func(*args, **kwargs)
  File "/home/rgrant/development/distarray/distarray/dist/context.py", line 133, in create_local
    distribution = Distribution(comm=comm, dim_data=dim_data)
  File "/home/rgrant/development/distarray/distarray/local/maps.py", line 51, in __init__
    "__main__", fname, loader, pkg_name)
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/rgrant/development/distarray/distarray/apps/engine.py", line 9, in <module>
    Engine()
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 41, in __init__
    val = self.parse_msg(msg)
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 80, in parse_msg
    ret = func(msg)
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 115, in func_call
    res = new_func(*args, **kwargs)
  File "/home/rgrant/development/distarray/distarray/dist/context.py", line 133, in create_local
    distribution = Distribution(comm=comm, dim_data=dim_data)
  File "/home/rgrant/development/distarray/distarray/local/maps.py", line 51, in __init__
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/rgrant/development/distarray/distarray/apps/engine.py", line 9, in <module>
    Engine()
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 41, in __init__
    val = self.parse_msg(msg)
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 80, in parse_msg
    ret = func(msg)
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 115, in func_call
    res = new_func(*args, **kwargs)
  File "/home/rgrant/development/distarray/distarray/dist/context.py", line 133, in create_local
    distribution = Distribution(comm=comm, dim_data=dim_data)
  File "/home/rgrant/development/distarray/distarray/local/maps.py", line 51, in __init__
    ret = func(msg)
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 115, in func_call
    res = new_func(*args, **kwargs)
  File "/home/rgrant/development/distarray/distarray/dist/context.py", line 133, in create_local
    res = new_func(*args, **kwargs)
  File "/home/rgrant/development/distarray/distarray/dist/context.py", line 133, in create_local
    self.comm = construct.init_comm(self.base_comm, self.grid_shape)
  File "/home/rgrant/development/distarray/distarray/local/construct.py", line 38, in init_comm
Traceback (most recent call last):
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/rgrant/development/distarray/distarray/apps/engine.py", line 9, in <module>
    Engine()
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 41, in __init__
    val = self.parse_msg(msg)
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 80, in parse_msg
    ret = func(msg)
Traceback (most recent call last):
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/rgrant/development/distarray/distarray/apps/engine.py", line 9, in <module>
    self.comm = construct.init_comm(self.base_comm, self.grid_shape)
  File "/home/rgrant/development/distarray/distarray/local/construct.py", line 38, in init_comm
    reorder=False)
  File "Comm.pyx", line 952, in mpi4py.MPI.Intracomm.Create_cart (src/mpi4py.MPI.c:69241)
Traceback (most recent call last):
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/rgrant/development/distarray/distarray/apps/engine.py", line 9, in <module>
    Engine()
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 41, in __init__
    val = self.parse_msg(msg)
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 80, in parse_msg
    ret = func(msg)
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 115, in func_call
    res = new_func(*args, **kwargs)
  File "/home/rgrant/development/distarray/distarray/dist/context.py", line 133, in create_local
    self.comm = construct.init_comm(self.base_comm, self.grid_shape)
  File "/home/rgrant/development/distarray/distarray/local/construct.py", line 38, in init_comm
    reorder=False)
  File "Comm.pyx", line 952, in mpi4py.MPI.Intracomm.Create_cart (src/mpi4py.MPI.c:69241)
    self.comm = construct.init_comm(self.base_comm, self.grid_shape)
  File "/home/rgrant/development/distarray/distarray/local/construct.py", line 38, in init_comm
    reorder=False)
  File "Comm.pyx", line 952, in mpi4py.MPI.Intracomm.Create_cart (src/mpi4py.MPI.c:69241)
    distribution = Distribution(comm=comm, dim_data=dim_data)
  File "/home/rgrant/development/distarray/distarray/local/maps.py", line 51, in __init__
    self.comm = construct.init_comm(self.base_comm, self.grid_shape)
  File "/home/rgrant/development/distarray/distarray/local/construct.py", line 38, in init_comm
    distribution = Distribution(comm=comm, dim_data=dim_data)
  File "/home/rgrant/development/distarray/distarray/local/maps.py", line 51, in __init__
    reorder=False)
  File "Comm.pyx", line 952, in mpi4py.MPI.Intracomm.Create_cart (src/mpi4py.MPI.c:69241)
mpi4py.MPI.Exception  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 115, in func_call
    res = new_func(*args, **kwargs)
  File "/home/rgrant/development/distarray/distarray/dist/context.py", line 133, in create_local
    distribution = Distribution(comm=comm, dim_data=dim_data)
    Engine()
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 41, in __init__
    val = self.parse_msg(msg)
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 80, in parse_msg
    distribution = Distribution(comm=comm, dim_data=dim_data)
  File "/home/rgrant/development/distarray/distarray/local/maps.py", line 51, in __init__
    self.comm = construct.init_comm(self.base_comm, self.grid_shape)
    reorder=False)
  File "Comm.pyx", line 952, in mpi4py.MPI.Intracomm.Create_cart (src/mpi4py.MPI.c:69241)
    self.comm = construct.init_comm(self.base_comm, self.grid_shape)
  File "/home/rgrant/development/distarray/distarray/local/construct.py", line 38, in init_comm
: Other MPI error, error stack:
PMPI_Cart_create(313).........: MPI_Cart_create(comm=0xc40c00cd, ndims=2, dims=0x472c2b0, periods=0x472b330, reorder=0, comm_cart=0x7f8c54715f70) failed
MPIR_Cart_create_impl(189)....:
MPIR_Cart_create(115).........:
MPIR_Comm_copy(967)...........:
MPIR_Get_contextid(521).......:
MPIR_Get_contextid_sparse(752): Too many communicators
  File "/home/rgrant/development/distarray/distarray/local/maps.py", line 51, in __init__
    self.comm = construct.init_comm(self.base_comm, self.grid_shape)
  File "/home/rgrant/development/distarray/distarray/local/construct.py", line 38, in init_comm
    ret = func(msg)
  File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 115, in func_call
mpi4py.MPI.Exception: Other MPI error, error stack:
PMPI_Cart_create(313).........: MPI_Cart_create(comm=0xc40800f5, ndims=2, dims=0x16daf80, periods=0x16e0860, reorder=0, comm_cart=0x7f6f0202ff70) failed
MPIR_Cart_create_impl(189)....:
MPIR_Cart_create(115).........:
MPIR_Comm_copy(967)...........:
MPIR_Get_contextid(521).......:
MPIR_Get_contextid_sparse(752): Too many communicators
  File "/home/rgrant/development/distarray/distarray/local/construct.py", line 38, in init_comm
    reorder=False)
  File "Comm.pyx", line 952, in mpi4py.MPI.Intracomm.Create_cart (src/mpi4py.MPI.c:69241)
mpi4py.MPI.Exception: Other MPI error, error stack:
PMPI_Cart_create(313).........: MPI_Cart_create(comm=0xc4060065, ndims=2, dims=0x118b0a0, periods=0x118e0f0, reorder=0, comm_cart=0x7f5486209f70) failed
MPIR_Cart_create_impl(189)....:
MPIR_Cart_create(115).........:
MPIR_Comm_copy(967)...........:
MPIR_Get_contextid(521).......:
MPIR_Get_contextid_sparse(752): Too many communicators
mpi4py.MPI.Exception: Other MPI error, error stack:
PMPI_Cart_create(313).........: MPI_Cart_create(comm=0xc405001d, ndims=2, dims=0x1d38040, periods=0x1d438b0, reorder=0, comm_cart=0x7fd9f8be9f58) failed
MPIR_Cart_create_impl(189)....:
MPIR_Cart_create(115).........:
MPIR_Comm_copy(967)...........:
MPIR_Get_contextid(521).......:
MPIR_Get_contextid_sparse(752): Too many communicators
mpi4py.MPI.Exception:     reorder=False)
  File "Comm.pyx", line 952, in mpi4py.MPI.Intracomm.Create_cart (src/mpi4py.MPI.c:69241)
    reorder=False)
  File "Comm.pyx", line 952, in mpi4py.MPI.Intracomm.Create_cart (src/mpi4py.MPI.c:69241)
    res = new_func(*args, **kwargs)
  File "/home/rgrant/development/distarray/distarray/dist/context.py", line 133, in create_local
Other MPI error, error stack:
PMPI_Cart_create(313).........: MPI_Cart_create(comm=0xc41500ef, ndims=2, dims=0x29bdb70, periods=0x29b7820, reorder=0, comm_cart=0x7fa0ac697f70) failed
MPIR_Cart_create_impl(189)....:
MPIR_Cart_create(115).........:
MPIR_Comm_copy(967)...........:
MPIR_Get_contextid(521).......:
MPIR_Get_contextid_sparse(752): Too many communicators
mpi4py.MPI.Exceptionmpi4py.MPI.Exception: Other MPI error, error stack:
PMPI_Cart_create(313).........: MPI_Cart_create(comm=0xc40700ad, ndims=2, dims=0x2ece820, periods=0x2ece840, reorder=0, comm_cart=0x7f718ce77f70) failed
MPIR_Cart_create_impl(189)....:
MPIR_Cart_create(115).........:
MPIR_Comm_copy(967)...........:
MPIR_Get_contextid(521).......:
MPIR_Get_contextid_sparse(752): Too many communicators
: Other MPI error, error stack:
PMPI_Cart_create(313).........: MPI_Cart_create(comm=0xc40e0015, ndims=2, dims=0x1a163f0, periods=0x1a16410, reorder=0, comm_cart=0x7ffa9190af70) failed
MPIR_Cart_create_impl(189)....:
MPIR_Cart_create(115).........:
MPIR_Comm_copy(967)...........:
MPIR_Get_contextid(521).......:
MPIR_Get_contextid_sparse(752): Too many communicators
mpi4py.MPI.Exception: Other MPI error, error stack:
PMPI_Cart_create(313).........: MPI_Cart_create(comm=0xc40b0085, ndims=2, dims=0x2cec520, periods=0x2ceb490, reorder=0, comm_cart=0x7f431c19bf70) failed
MPIR_Cart_create_impl(189)....:
MPIR_Cart_create(115).........:
MPIR_Comm_copy(967)...........:
MPIR_Get_contextid(521).......:
MPIR_Get_contextid_sparse(752): Too many communicators
    distribution = Distribution(comm=comm, dim_data=dim_data)
  File "/home/rgrant/development/distarray/distarray/local/maps.py", line 51, in __init__
    self.comm = construct.init_comm(self.base_comm, self.grid_shape)
  File "/home/rgrant/development/distarray/distarray/local/construct.py", line 38, in init_comm
    reorder=False)
  File "Comm.pyx", line 952, in mpi4py.MPI.Intracomm.Create_cart (src/mpi4py.MPI.c:69241)
mpi4py.MPI.Exception: Other MPI error, error stack:
PMPI_Cart_create(313).........: MPI_Cart_create(comm=0xc40a003d, ndims=2, dims=0x11f30b0, periods=0x11f84f0, reorder=0, comm_cart=0x7fbf2c609f70) failed
MPIR_Cart_create_impl(189)....:
MPIR_Cart_create(115).........:
MPIR_Comm_copy(967)...........:
MPIR_Get_contextid(521).......:
MPIR_Get_contextid_sparse(752): Too many communicators

It got much further, but gave a different error.

Then, I tried out a stress test with the numpy fancy-indexing kernel using an IPythonContext:

$ python benchmark_julia.py -o11000_ipython.json -r6 --kernel=fancy 11000
(n/n_runs: time) ('Start', 'End', 'Dist', 'Resolution', 'c', 'Engines', 'Iters')
(1/294: 349.567s) (1406435193.460827, 1406435543.028039, 'numpy', 11000, '(-0.045+0.45j)', 1, [3696797358])
(2/294: 345.118s) (1406435543.735211, 1406435888.852837, 'b-n', 11000, '(-0.045+0.45j)', 1, [3696797358])
(3/294: 347.838s) (1406435889.605796, 1406436237.443562, 'c-n', 11000, '(-0.045+0.45j)', 1, [3696797358])
(4/294: 359.565s) (1406436238.182001, 1406436597.746668, 'b-b', 11000, '(-0.045+0.45j)', 1, [3696797358])
(5/294: 354.985s) (1406436598.521989, 1406436953.507226, 'c-c', 11000, '(-0.045+0.45j)', 1, [3696797358])
(6/294: 180.324s) (1406436954.102975, 1406437134.426741, 'b-n', 11000, '(-0.045+0.45j)', 2, [1848009514, 1848787844])
(7/294: 183.050s) (1406437134.911213, 1406437317.960948, 'c-n', 11000, '(-0.045+0.45j)', 2, [1848392664, 1848404694])
(8/294: 184.815s) (1406437318.429465, 1406437503.244464, 'b-b', 11000, '(-0.045+0.45j)', 2, [1848100010, 1848697348])
(9/294: 181.443s) (1406437503.726369, 1406437685.168943, 'c-c', 11000, '(-0.045+0.45j)', 2, [1848396939, 1848400419])
(10/294: 158.055s) (1406437685.604622, 1406437843.659871, 'b-n', 11000, '(-0.045+0.45j)', 3, [685995820, 2324806074, 685995464])
(11/294: 124.062s) (1406437844.07946, 1406437968.141083, 'c-n', 11000, '(-0.045+0.45j)', 3, [1232260508, 1232276269, 1232260581])
(12/294: 141.303s) (1406437968.539321, 1406438109.841954, 'b-b', 11000, '(-0.045+0.45j)', 3, [748452256, 2199892627, 748452475])
(13/294: 128.234s) (1406438110.242824, 1406438238.477027, 'c-c', 11000, '(-0.045+0.45j)', 3, [1232262847, 1232271595, 1232262916])
(14/294: 107.972s) (1406438238.862006, 1406438346.834492, 'b-n', 11000, '(-0.045+0.45j)', 4, [289093817, 1558915697, 1559344553, 289443291])
(15/294: 97.990s) (1406438347.255682, 1406438445.245199, 'c-n', 11000, '(-0.045+0.45j)', 4, [924200092, 924202310, 924192572, 924202384])
(16/294: 101.836s) (1406438445.615707, 1406438547.451687, 'b-b', 11000, '(-0.045+0.45j)', 4, [655791340, 1192218174, 1192308670, 656479174])
(17/294: 103.377s) (1406438547.827442, 1406438651.204403, 'c-c', 11000, '(-0.045+0.45j)', 4, [924201433, 924191231, 924195506, 924209188])
(18/294: 91.840s) (1406438652.991555, 1406438744.831066, 'b-n', 11000, '(-0.045+0.45j)', 5, [127608174, 953151016, 1534681258, 953539801, 127817109])
(19/294: 88.682s) (1406438745.43266, 1406438834.114253, 'c-n', 11000, '(-0.045+0.45j)', 5, [739371975, 739361284, 739351486, 739351267, 739361346])
(20/294: 91.953s) (1406438834.955011, 1406438926.907979, 'b-b', 11000, '(-0.045+0.45j)', 5, [60876925, 1158242038, 1257966037, 1158620346, 61092012])
(21/294: 94.120s) (1406438927.43213, 1406439021.5518, 'c-c', 11000, '(-0.045+0.45j)', 5, [739364921, 739355182, 739361005, 739360920, 739355330])
(22/294: 83.065s) (1406439022.584114, 1406439105.6491, 'b-n', 11000, '(-0.045+0.45j)', 6, [63135170, 623389934, 1163042463, 1162292639, 622223585, 62713567])
(23/294: 72.385s) (1406439106.217274, 1406439178.601975, 'c-n', 11000, '(-0.045+0.45j)', 6, [616129929, 616143599, 616130065, 616130579, 616132670, 616130516])
(24/294: 84.711s) (1406439179.159553, 1406439263.870242, 'b-b', 11000, '(-0.045+0.45j)', 6, [213752450, 1099763033, 534494031, 534699806, 1100129594, 213958444])
(25/294: 78.355s) (1406439264.387081, 1406439342.742386, 'c-c', 11000, '(-0.045+0.45j)', 6, [616130702, 616131119, 616130843, 616132145, 616140476, 616132073])
(26/294: 75.927s) (1406439343.664741, 1406439419.592154, 'b-n', 11000, '(-0.045+0.45j)', 7, [30962137, 396980025, 854396551, 1134174945, 853437138, 396202636, 30643926])
(27/294: 62.778s) (1406439420.287868, 1406439483.06595, 'c-n', 11000, '(-0.045+0.45j)', 7, [528119782, 528126679, 528126607, 528119844, 528106868, 528090927, 528106651])
(28/294: 73.182s) (1406439483.568793, 1406439556.750723, 'b-b', 11000, '(-0.045+0.45j)', 7, [6787090, 446295211, 952251426, 887842857, 952018789, 444850328, 6751657])
(29/294: 69.557s) (1406439557.260535, 1406439626.817696, 'c-c', 11000, '(-0.045+0.45j)', 7, [528123032, 528126337, 528126317, 528123177, 528103166, 528092169, 528103160])
(30/294: 71.463s) (1406439627.654691, 1406439699.118083, 'b-n', 11000, '(-0.045+0.45j)', 8, [12248655, 276845162, 640458941, 918456756, 918692944, 640651609, 277129577, 12313714])
(31/294: 61.419s) (1406439701.319855, 1406439762.738971, 'c-n', 11000, '(-0.045+0.45j)', 8, [462101201, 462097709, 462096354, 462104604, 462098891, 462104601, 462096218, 462097780])
(32/294: 74.349s) (1406439763.250448, 1406439837.599634, 'b-b', 11000, '(-0.045+0.45j)', 8, [83533222, 572258118, 1008374595, 183843579, 183621542, 1008687128, 572715690, 83763484])
(33/294: 63.601s) (1406439838.165366, 1406439901.7661, 'c-c', 11000, '(-0.045+0.45j)', 8, [462098824, 462095653, 462102609, 462095578, 462095605, 462104487, 462099901, 462104701])
(34/294: 60.616s) (1406439903.164532, 1406439963.780276, 'b-n', 11000, '(-0.045+0.45j)', 9, [4925883, 183016549, 499111699, 715249476, 896278422, 714337094, 497544322, 181478870, 4855043])
(35/294: 60.553s) (1406439964.579545, 1406440025.132668, 'c-n', 11000, '(-0.045+0.45j)', 9, [410741906, 410737132, 410741974, 410750876, 410769531, 410767668, 410767726, 410769606, 410750939])
(36/294: 69.629s) (1406440026.029944, 1406440095.659217, 'b-b', 11000, '(-0.045+0.45j)', 9, [8579385, 427601933, 249814502, 490058575, 1344688900, 490058599, 249814296, 427601794, 8579374])
(37/294: 51.912s) (1406440096.418413, 1406440148.330354, 'c-c', 11000, '(-0.045+0.45j)', 9, [410751485, 410754468, 410754555, 410756806, 410762587, 410756876, 410754556, 410754540, 410751485])
(38/294: 55.903s) (1406440149.653172, 1406440205.556156, 'b-n', 11000, '(-0.045+0.45j)', 10, [3770571, 123837603, 366462193, 586688823, 767250324, 767430934, 586813991, 366725810, 124038227, 3778882])
(39/294: 56.107s) (1406440206.719589, 1406440262.826881, 'c-n', 11000, '(-0.045+0.45j)', 10, [369682922, 369681132, 369674794, 369676626, 369680155, 369689053, 369680152, 369676692, 369674641, 369681191])
(40/294: 67.764s) (1406440263.756064, 1406440331.519789, 'b-b', 11000, '(-0.045+0.45j)', 10, [30438885, 350901601, 628804333, 807360406, 30504289, 30438040, 807340437, 629161704, 351259940, 30587723])
(41/294: 66.957s) (1406440332.213713, 1406440399.170588, 'c-c', 11000, '(-0.045+0.45j)', 10, [369679372, 369678018, 369678558, 369678548, 369678168, 369685549, 369677164, 369682447, 369682372, 369677162])
(42/294: 53.369s) (1406440400.687472, 1406440454.056887, 'b-n', 11000, '(-0.045+0.45j)', 11, [3013488, 85985841, 287115608, 486175090, 614021971, 743483469, 614182540, 486351466, 287295517, 86151974, 3020394])
(43/294: 54.508s) (1406440455.090445, 1406440509.598092, 'c-n', 11000, '(-0.045+0.45j)', 11, [336084165, 336088852, 336076254, 336070636, 336060695, 336060122, 336060062, 336060692, 336070703, 336076330, 336088847])
(44/294: 54.738s) (1406440510.267967, 1406440565.006075, 'b-b', 11000, '(-0.045+0.45j)', 11, [2319295, 29929569, 350035841, 581368098, 597416456, 574113506, 597308476, 581543631, 350408188, 30030022, 2324276])
(45/294: 57.531s) (1406440565.767961, 1406440623.298978, 'c-c', 11000, '(-0.045+0.45j)', 11, [336079361, 336072968, 336073255, 336076735, 336074614, 336061650, 336061639, 336074334, 336076731, 336073185, 336072886])
(46/294: 64.956s) (1406440624.940203, 1406440689.896576, 'b-n', 11000, '(-0.045+0.45j)', 12, [2469761, 60665409, 226308308, 397081626, 508838127, 654204336, 653761606, 508531033, 396542913, 225680672, 60262263, 2451304])
(47/294: 54.343s) (1406440690.601154, 1406440744.944298, 'c-n', 11000, '(-0.045+0.45j)', 12, [308066895, 308071799, 308063097, 308062891, 308066229, 308062823, 308063034, 308071800, 308066968, 308067688, 308066441, 308067693])
(48/294: 60.943s) (1406440745.737092, 1406440806.679765, 'b-b', 11000, '(-0.045+0.45j)', 12, [1749744, 112879631, 504299595, 67066850, 198491188, 963728487, 963798300, 198788099, 66913832, 504337128, 112992390, 1752114])
(49/294: 50.527s) (1406440807.33579, 1406440857.862712, 'c-c', 11000, '(-0.045+0.45j)', 12, [308063205, 308067084, 308065765, 308064454, 308068091, 308068674, 308070906, 308068598, 308063133, 308064382, 308065839, 308067227])
(50/294: 349.402s) (1406440962.815184, 1406441312.21672, 'numpy', 11000, '(-0.045+0.45j)', 1, [3696797358])
Traceback (most recent call last):
  File "benchmark_julia.py", line 362, in <module>
    cli(sys.argv)
  File "benchmark_julia.py", line 357, in cli
    kernel=fn_from_kernel[args.kernel])
  File "benchmark_julia.py", line 289, in do_julia_runs
    print(prog_fmt.format(n, n_runs, result[1] - result[0]), result)
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/contextlib.py", line 154, in __exit__
    self.thing.close()
  File "/home/rgrant/development/distarray/distarray/dist/context.py", line 705, in close
    self.cleanup()
  File "/home/rgrant/development/distarray/distarray/dist/context.py", line 702, in cleanup
    prefix=self.context_key)
  File "/home/rgrant/development/distarray/distarray/dist/ipython_cleanup.py", line 21, in cleanup
    view.apply_sync(_cleanup, module_name, prefix)
  File "<string>", line 2, in apply_sync
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/view.py", line 75, in spin_after
    ret = f(self, *args, **kwargs)
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/view.py", line 247, in apply_sync
    return self._really_apply(f, args, kwargs, block=True)
  File "<string>", line 2, in _really_apply
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/view.py", line 66, in sync_results
    ret = f(self, *args, **kwargs)
  File "<string>", line 2, in _really_apply
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/view.py", line 51, in save_ids
    ret = f(self, *args, **kwargs)
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/view.py", line 577, in _really_apply
    return ar.get()
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/asyncresult.py", line 118, in get
    raise self._exception
IPython.parallel.error.CompositeError: one or more exceptions from call to method: _cleanup
[Engine Exception]EngineError: Engine 0 died while running task 'f167978c-bab6-433d-a342-95cb22986faa'
[Engine Exception]EngineError: Engine 1 died while running task '23c16002-0e2e-49b5-bc0f-3fcbc99b85bb'
[Engine Exception]EngineError: Engine 2 died while running task '15c2bc93-e5c9-4bd1-9dda-639a53ba4ccb'
[Engine Exception]EngineError: Engine 3 died while running task '4d92d260-c7da-4d41-8e13-ebec83aaabe4'
.... 8 more exceptions ...
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/home/rgrant/development/distarray/distarray/dist/ipython_cleanup.py", line 30, in cleanup_all
    v = c[:]
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/client.py", line 923, in __getitem__
    return self.direct_view(key)
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/client.py", line 1363, in direct_view
    targets = self._build_targets(targets)[1]
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/client.py", line 567, in _build_targets
    raise error.NoEnginesRegistered("Can't build targets without any engines")
NoEnginesRegistered: Can't build targets without any engines
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/home/rgrant/development/distarray/distarray/dist/ipython_cleanup.py", line 71, in clear_all
    v = c[:]
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/client.py", line 923, in __getitem__
    return self.direct_view(key)
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/client.py", line 1363, in direct_view
    targets = self._build_targets(targets)[1]
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/client.py", line 567, in _build_targets
    raise error.NoEnginesRegistered("Can't build targets without any engines")
NoEnginesRegistered: Can't build targets without any engines
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/home/rgrant/development/distarray/distarray/dist/ipython_cleanup.py", line 71, in clear_all
    v = c[:]
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/client.py", line 923, in __getitem__
    return self.direct_view(key)
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/client.py", line 1363, in direct_view
    targets = self._build_targets(targets)[1]
  File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/client.py", line 567, in _build_targets
    raise error.NoEnginesRegistered("Can't build targets without any engines")
IPython.parallel.error.NoEnginesRegistered: Can't build targets without any engines

Dominant language: Python
Stars: 5
Forks: 1
PR merge metrics: No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

Similar issues

Report invalid UTF-8 case manifests through CaseManifestError Open

bug ci good first issue

Difficulty 2/5 1-3 hours Newbie friendliness 88/100

wavefnd/Wave#688 · Opened Sep 22, 2026
A support participant can take over an offered mic and silence the booth Open

Difficulty 2/5 1-3 hours Newbie friendliness 76/100

fossasia/voxbento#515 · Opened Sep 22, 2026
Add documentation on how to use tokens Open

documentation

Difficulty 2/5 Half a day Newbie friendliness 62/100

inmanta/inmanta-core#10835 · Opened Sep 22, 2026
YN-1100: Nuke: Set viewer colorspace fails on copy knob None value Open

sponsored

Difficulty 2/5 1-3 hours Newbie friendliness 65/100

ynput/ayon-nuke#375 · Opened Sep 22, 2026
opensubtitlescom: moviehash never sent when opensubtitles (.org) is not in the provider list Open

Difficulty 2/5 1-3 hours Newbie friendliness 86/100

Diaoul/subliminal#1382 · Opened Sep 22, 2026

More Python issues