mozilla/overscripted
Can we use array extensions to speed up string processing with dask
オープン
#36 opened on 2019/03/11
good first issueresearch question
Repository metrics
- Stars
- (73 個のスター)
- PR merge metrics
- (PR metrics pending)
説明
Much of the analysis done on this dataset uses dask. Dask is excellent for distributed numerical computing but seems to struggle with strings.
Array extensions unfortunately can't be serialized https://github.com/pandas-dev/pandas/issues/20612
https://github.com/xhochy/fletcher is an array extension that adds string processing functionality.
Doing some standard tasks for this dataset like: collecting all domains, or number of script domains per location domain compare the performance of spark, dask, and dask with extension arrays.