Significant slowdown on large files
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 35/100
- Issue 类型
- 重构
- 描述清晰度
- 基本清楚
- 活跃度
- 停滞
- 技术栈
- cpp
- 领域
- performance
调研方向
Start by locating happly's list-property storage and getListProperty API, then profile parsing of the Lucy model from the Stanford repo to confirm the vector-allocation cost. Done means reducing large-file slowdown while preserving the existing list-property APIs and adding the proposed preallocated-list access without breaking backward compatibility.
由索引模型根据 Issue 内容生成。
描述
The current version of happly is quite slow for large files compared to an home-brewed solution I cooked up. The profiler suggest that the problem is allocating many small vectors in list properties. On the Lucy model from the Stanford repo, happly takes about 16 seconds of which 7 are just vector allocs. My home-brewed solution takes half that time.
I propose the following changes:
- change the storage of list props from
vector<vector<T>> datato three vectors for start, count and datastd::vector<size_t> start; std::vector<uint8_t> count; vector<T> data;, where data has the concatenated list of elements, start has the starting index for each list and count contains the lists sizes - in a backward compatible manner, add
getListProperty(vector<array<T, N>>& data, vector<uint8_t>& count)to read the data in preallocated lists; maintain previous versions for backward compatibility
If this sounds good, I may even take a crack at it, but only if this feels right.
- 主要语言
- C++
- 星标
- 406
- 派生
- 74
- PR 合并指标
- 30 天内没有已合并 PR
贡献指南
这个仓库没有索引到贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
nmwsharp/happly 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 76/100
-
难度 3/5 1-2 天 新手友好度 56/100
-
难度 3/5 1-2 天 新手友好度 48/100
-
难度 3/5 1-2 天 新手友好度 55/100
-
难度 2/5 1-3 小时 新手友好度 45/100
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 75/100
flutter-webrtc/flutter-webrtc#2206 ·
-
难度 2/5 1-3 小时 新手友好度 70/100
google-ai-edge/LiteRT-LM#3739 ·
-
Component: GLib
难度 2/5 1-3 小时 新手友好度 70/100
-
难度 2/5 1-3 小时 新手友好度 65/100
brave/brave-browser#59300 ·
-
Mute ydb/tests/functional/dstool/test_canonical_requests.py.Test.test_group_take_snapshot in main 未关闭ai_reviewed
难度 2/5 1-3 小时 新手友好度 70/100
ydb-platform/ydb#53974 · 3 条评论 ·