mozilla/overscripted

Can we use array extensions to speed up string processing with dask

开放

#36 创建于 2019年3月11日

 (13 条评论) (0 个反应) (0 位负责人)Jupyter Notebook (48 个派生)auto 404
good first issueresearch question

仓库指标

星标
 (73 个星标)
PR 合并指标
 (PR 指标待抓取)

描述

Much of the analysis done on this dataset uses dask. Dask is excellent for distributed numerical computing but seems to struggle with strings.

Array extensions unfortunately can't be serialized https://github.com/pandas-dev/pandas/issues/20612

https://github.com/xhochy/fletcher is an array extension that adds string processing functionality.

Doing some standard tasks for this dataset like: collecting all domains, or number of script domains per location domain compare the performance of spark, dask, and dask with extension arrays.

贡献者指南