tableFromIPC incorrectly types columns in schema when there are duplicate column names
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 45/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- typescript
- Domain
- data
Research direction
Start by reproducing the issue with the supplied base64 Arrow IPC data and the tableFromIPC entry point. Compare the returned schema with getChildAt(i).type for duplicate id columns, using the pyarrow output as the expected result. Done means the two duplicate fields retain their individual types in the table schema.
Written by the indexing model from the issue text.
Description
Describe the bug, including details regarding any error messages, version, and platform.
Version: apache-arrow@21.0.0 (also tested and noticed the issue in 17.0.0)
I have a use case where I have an arrow file that has two columns with the same name, but different types. In this example, there are two "id" columns, but one is an Int while the other is a string.
When I load this arrow file using tableFromIPC, the table schema marks both columns as strings.
Simple base64 arrow file used in this example:
/////7ABAAAQAAAAAAAKAA4ABgANAAgACgAAAAAABAAQAAAAAAEKAAwAAAAIAAQACgAAAAgAAAAIAAAAAAAAAAIAAADAAAAABAAAAFr///8UAAAAiAAAAIwAAAAAAAAFiAAAAAIAAABAAAAABAAAABT///8IAAAAFAAAAAgAAAAic3RyaW5nIgAAAAAXAAAAU3Bhcms6RGF0YVR5cGU6SnNvblR5cGUATP///wgAAAAQAAAABgAAAFNUUklORwAAFgAAAFNwYXJrOkRhdGFUeXBlOlNxbE5hbWUAAAAAAAAEAAQABAAAAAIAAABpZAAAAAASABgAFAAAABMADAAAAAgABAASAAAAFAAAAIwAAACUAAAAAAAAApgAAAACAAAARAAAAAQAAADM////CAAAABAAAAAGAAAAImxvbmciAAAXAAAAU3Bhcms6RGF0YVR5cGU6SnNvblR5cGUACAAMAAgABAAIAAAACAAAABAAAAAGAAAAQklHSU5UAAAWAAAAU3Bhcms6RGF0YVR5cGU6U3FsTmFtZQAAAAAAAAgADAAIAAcACAAAAAAAAAFAAAAAAgAAAGlkAAD/////yAAAABQAAAAAAAAADAAWAAYABQAIAAwADAAAAAADBAAYAAAAKAAAAAAAAAAAAAoAGAAMAAQACAAKAAAAbAAAABAAAAABAAAAAAAAAAAAAAAFAAAAAAAAAAAAAAABAAAAAAAAAAgAAAAAAAAACAAAAAAAAAAQAAAAAAAAAAEAAAAAAAAAGAAAAAAAAAAIAAAAAAAAACAAAAAAAAAAAgAAAAAAAAAAAAAAAgAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAIAAAAxMAAAAAAAAP////8AAAAA
Below are the details of the Table object returned by tableFromIPC. Notice how in the schema, both id columns are marked as Utf8, but the column details show the difference in column types.
=== Arrow Table Information ===
Rows: 1
Columns: 2
Schema: Schema<{ 0: id: Utf8, 1: id: Utf8 }>
=== Table Contents ===
[
{"id": 0, "id": "10"}
]
=== Column Details ===
Column 0: undefined (Int64)
Column 1: undefined (Utf8)
Code to generate the above result
const arrowBuffer = base64ToUint8Array(base64String);
// Parse the Arrow IPC data into a table
const table = tableFromIPC(arrowBuffer);
console.log('\n=== Arrow Table Information ===');
console.log(`Rows: ${table.numRows}`);
console.log(`Columns: ${table.numCols}`);
console.log(`Schema: ${table.schema}`);
console.log('\n=== Table Contents ===');
console.log(table.toString());
console.log('\n=== Column Details ===');
for (let i = 0; i < table.numCols; i++) {
const column = table.getChildAt(i);
console.log(`Column ${i}: ${column.name} (${column.type})`);
}
I have tested this same arrow file with pyarrow, and it shows the expected result:
Schema:
id: int64 not null
-- field metadata --
Spark:DataType:SqlName: 'BIGINT'
Spark:DataType:JsonType: '"long"'
id: string not null
-- field metadata --
Spark:DataType:SqlName: 'STRING'
Spark:DataType:JsonType: '"string"'
Batch 0:
pyarrow.RecordBatch
id: int64 not null
id: string not null
----
id: [0]
id: ["10"]
- Dominant language
- TypeScript
- Stars
- 112
- Forks
- 23
- Avg merge
- 17h 55m
- Merged PRs (30d)
- 10
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/arrow-js
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
-
Difficulty 3/5 1-2 days Newbie friendliness 55/100
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
Difficulty 3/5 1-2 days Newbie friendliness 50/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
mksglu/context-mode#1200 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
jaegertracing/jaeger-ui#4506 ·
-
area:desktop area:ui bug platform:macos
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
anthropics/claude-code#96687 ·
-
good first issue
Difficulty 1/5 Under an hour Newbie friendliness 95/100
AOSSIE-Org/DebateAI#582 · 2 comments ·