mozilla/overscripted
Can we use array extensions to speed up string processing with dask
開放
#36 建立於 2019年3月11日
good first issueresearch question
倉庫指標
- 星標
- (73 顆星)
- PR 合併指標
- (PR 指標待抓取)
描述
Much of the analysis done on this dataset uses dask. Dask is excellent for distributed numerical computing but seems to struggle with strings.
Array extensions unfortunately can't be serialized https://github.com/pandas-dev/pandas/issues/20612
https://github.com/xhochy/fletcher is an array extension that adds string processing functionality.
Doing some standard tasks for this dataset like: collecting all domains, or number of script domains per location domain compare the performance of spark, dask, and dask with extension arrays.