新增大 Excel 轻量级解析路径,解决大表格 OOM
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- python
- Domain
- data, performance
Research direction
Start by locating load_workbook() and _find_true_data_bounds, then trace how merged ranges affect the worksheet bounds and cell creation. Reproduce the large-sheet case described in the issue and compare the existing path with the proposed SAX streaming path. Done means large Excel files can be parsed without the current 12GB+ memory failure and with the stated lower peak usage.
Written by the indexing model from the issue text.
Description
描述
处理超大 Excel(如 33MB XML sheet、74 万单元格)时,openpyxl 全量加载创建百万级 Cell 对象,内存占用超过 12GB,进程被 OOM 杀死。本 PR 新增轻量级解析路径(SAX 流式解析),内存峰值从 12GB+ 降至 2.2GB。
根因
load_workbook() 为每个单元格(含空位)创建 Cell 对象 -> 百万级对象 -> 12GB
异常文件含 18 万行合并区域(18.6 万个),拉大洪水填充边界
_find_true_data_bounds 将合并区域纳入边界,max_row=186365,洪水填充遍历 74 万空单元格
- Dominant language
- Python
- Stars
- 233
- Forks
- 41
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from RapidAI/RapidDoc
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
-
paddleocr-vl 1.6 支持Open
Difficulty 4/5 3-5 days Newbie friendliness 35/100
All issues in RapidAI/RapidDoc
Similar issues
-
comp/tools P2 type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
NousResearch/hermes-agent#142908 · 1 comment ·
-
[oblt-aw][autodoc] Fix malformed headings in new Buildkite docs pagesPossibly taken @github-actions claimed this today. Opendocumentation
Difficulty 1/5 Under an hour Newbie friendliness 83/100
Maintainers usually reply within 1 day
-
[Bug]: fileUploadMatrix.test.ts URL case does a real DNS lookup and fails offlinePossibly taken @r2jaff claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
ComposioHQ/composio#5054 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
pytorch/tensordict#3286 ·
Maintainers usually reply within 1 day