SciRuby/daru

CSV reading performance issue.

Open

#337 建立於 2017年4月26日

在 GitHub 查看
 (17 留言) (0 反應) (0 負責人)Ruby (140 fork)github user discovery
difficulty: experthelp wantedperformance

倉庫指標

Star
 (1,060 star)
PR 合併指標
 (PR 指標待抓取)

描述

I just tried to read a csv file of size 42.6 MB which consists of 550241 records. When I tried with daru here is following statistics

# analysis.rb
require 'daru'
df = Daru::DataFrame.from_csv('./IndiaAffectedWaterQualityAreas.csv')
p df.first(2)

$ time ruby analysis.rb

real	0m48.878s
user	0m47.898s
sys	0m0.798s

if I do the same with python then it shows as follows

# analysis.py
import numpy as np # linear algebra
import pandas as pd
data = pd.read_csv('./IndiaAffectedWaterQualityAreas.csv', encoding = "ISO-8859-1")
print(data.tail(25))

$ time python analysis.py

real	0m1.649s
user	0m1.439s
sys	0m0.195s

Since Github doesn't allow a file to upload more than 10 MB so I couldn't upload. But you can get the data from https://www.kaggle.com/venkatramakrishnan/india-water-quality-data

貢獻者指南