mattn/go-sqlite3

Next and Scan appear to be relatively slow?

Open

#379 创建于 2017年2月7日

在 GitHub 查看
 (10 评论) (3 反应) (0 负责人)C (7,000 star) (1,105 fork)batch import
help wantedresearch

描述

Howdy. I'm not sure this really merits a Issues or if it would be better as a conversation some place. In using sqlite3 for a project and I noticed that it's quite a bit slower to select a large number of rows as compared to doing the same via their CLI client. ~4-5x slower. A little profiling shows that the time is mostly spent in Next and Scan. For small row sets it's not a big deal but I'm testing up to ~500K rows.

I suppose my starting question is how much overhead is expected going back and forth between Go and the C API? If it's expected to be fairly fast/low-overhead, any suspicions on where I might start looking?

Here's a sketch of approximately what the code is doing:

https://gist.github.com/mhat/ded97fe58922b4af9d03f375b465c339

There are two variations, one uses the new ScanTypes while the other is cheating with a list of string pointers. Neither appreciably moved the needly forScan.

贡献者指南

技术栈
gosqlite
领域
database
议题类型
performance
难度面向新贡献者的预计实现难度，1 表示很小改动，5 表示专家级工作。
4
预计时间有经验贡献者完成调查、实现、测试并准备 pull request 的粗略时间范围。
1-2 days
活动状态议题当前的可参与程度：新鲜、活跃、陈旧、阻塞或等待维护者输入。
stale
清晰度议题是否清楚说明期望改动、验收标准和下一步。
needs investigation
前置要求
Go programmingSQLite C APIProfiling with pprof
新手友好度1-100 的估计分数，表示该议题对首次贡献者的友好程度。
15
研究方向
Profile the Next and Scan functions in the go sqlite3 driver. Examine the Go C callback mechanism and compare with native sqlite3 CLI. The attached gist shows sample code. No linked PRs or maintainer comments found.