Support large arrays
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 20/100
Research direction
Start by reviewing the ArrowBuffer and ArrayData APIs described in the issue, then read the related issues and pull request for prior design context. Define how large arrays could be supported without violating existing compatibility expectations, and document a concrete scope and completion criteria before implementation.
Written by the indexing model from the issue text.
Description
Describe the enhancement requested
It's hard for the C# Arrow library to support large arrays. Commonly-used types like Array, Span and Memory use a 32-bit signed integer for the length, meaning that they can't have more than ~2 million elements in them. The ArrowBuffer structure that is the foundation of the core ArrayData class exposes these types directly. This may make it impossible to support large arrays in a strictly backwards-compatible fashion.
Some previously-opened items related to this feature are:
https://github.com/apache/arrow/issues/23776
https://github.com/apache/arrow/issues/38086
https://github.com/apache/arrow/pull/38619
https://github.com/apache/arrow/issues/45030
- Dominant language
- C#
- Stars
- 41
- Forks
- 31
- Avg merge
- 8h 2m
- Merged PRs (30d)
- 15
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/arrow-dotnet
-
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
apache/arrow-dotnet#410 · 4 comments ·
Maintainers usually reply within 1 day
-
ArrowArrayConcatenator: null arrays are refused, and dictionary arrays lose their dictionaryPossibly taken @CurtHagenlocher claimed this 16 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 68/100
apache/arrow-dotnet#446 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 38/100
apache/arrow-dotnet#409 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
apache/arrow-dotnet#397 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 55/100
apache/arrow-dotnet#378 ·
Maintainers usually reply within 1 day
All issues in apache/arrow-dotnet
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
Maintainers usually reply within 1 day
-
area:re1 bounty help wanted needs-game
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
StackExchange/StackExchange.Redis#3274 ·
Maintainers usually reply within 1 day
-
affected/v17 affected/v18 area/frontend type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
umbraco/Umbraco-CMS#24186 ·
Maintainers usually reply within 2 days