Repository metrics
- Stars
- (852 stars)
- PR merge metrics
- (PR metrics pending)
Description
We should return to reevaluate the performance implications of broadcasting input data to all workers, all the time in the DaskEngine impl. I would assume Dask would block on completion of the data reaching the workers before it moves on to using the data in its task graphs, so its conceivable that we might be forcing Dask to slow down.
Studying this might be somewhat challenging, as a real world study would probably involve having some Dask cluster that exists where the workers are separated over a network from the scheduler so that there's some latency associated with large amounts of data moving to workers. But ideally, the ideal outcome of this issue involves a test using large dataset, a cluster with multiple workers and executing some not particularly compute expensive computation on the data. We'd like to test the timing difference of broadcast=True vs. broadcast=False.