SciRuby/daru

CSV reading performance issue.

Open

#337 创建于 2017年4月26日

在 GitHub 查看
 (17 评论) (0 反应) (0 负责人)Ruby (140 fork)github user discovery
difficulty: experthelp wantedperformance

仓库指标

Star
 (1,060 star)
PR 合并指标
 (PR 指标待抓取)

描述

I just tried to read a csv file of size 42.6 MB which consists of 550241 records. When I tried with daru here is following statistics

# analysis.rb
require 'daru'
df = Daru::DataFrame.from_csv('./IndiaAffectedWaterQualityAreas.csv')
p df.first(2)

$ time ruby analysis.rb

real	0m48.878s
user	0m47.898s
sys	0m0.798s

if I do the same with python then it shows as follows

# analysis.py
import numpy as np # linear algebra
import pandas as pd
data = pd.read_csv('./IndiaAffectedWaterQualityAreas.csv', encoding = "ISO-8859-1")
print(data.tail(25))

$ time python analysis.py

real	0m1.649s
user	0m1.439s
sys	0m0.195s

Since Github doesn't allow a file to upload more than 10 MB so I couldn't upload. But you can get the data from https://www.kaggle.com/venkatramakrishnan/india-water-quality-data

贡献者指南