mozilla/overscripted

Can we use array extensions to speed up string processing with dask

開放

#36 建立於 2019年3月11日

 (13 則留言) (0 個反應) (0 位負責人)Jupyter Notebook (48 個分叉)auto 404
good first issueresearch question

倉庫指標

星標
 (73 顆星)
PR 合併指標
 (PR 指標待抓取)

描述

Much of the analysis done on this dataset uses dask. Dask is excellent for distributed numerical computing but seems to struggle with strings.

Array extensions unfortunately can't be serialized https://github.com/pandas-dev/pandas/issues/20612

https://github.com/xhochy/fletcher is an array extension that adds string processing functionality.

Doing some standard tasks for this dataset like: collecting all domains, or number of script domains per location domain compare the performance of spark, dask, and dask with extension arrays.

貢獻者指南