RecordBuilder should confirm non-nullable fields do not contain null rows when building Record

Open
#372 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
go
Domain
data

Research direction

Start at array.NewRecordBuilder and RecordBuilder.NewRecord, using the reported schema and null-containing field as the reproduction case. Confirm the intended behavior against the Rust example and ensure that a non-nullable field containing null rows is rejected when building the record.

Written by the indexing model from the issue text.

Description

Type: bug
Describe the bug, including details regarding any error messages, version, and platform.

I am able to construct an arrow record, where a field should be non-nullable, but it contains null rows. I'm wondering if maybe RecordBuilder.NewRecord() should confirm that the fields's arrays do not contain nulls if the field is specified as Nullable: false in the schema?

version = v18.2.0
platform = arm mac

package main

import (
	"log"

	"github.com/apache/arrow-go/v18/arrow"
	"github.com/apache/arrow-go/v18/arrow/array"
	"github.com/apache/arrow-go/v18/arrow/memory"
)

func main() {
	allocator := memory.NewGoAllocator()
	schema := arrow.NewSchema([]arrow.Field{
	        // should be not null:
		{Name: "a", Type: arrow.PrimitiveTypes.Int32, Nullable: false}, 
	}, nil)
	recordBuilder := array.NewRecordBuilder(allocator, schema)

	recordBuilder.Field(0).(*array.Int32Builder).Append(1)
	recordBuilder.Field(0).(*array.Int32Builder).Append(2)
	recordBuilder.Field(0).AppendNull()
	recordBuilder.Field(0).(*array.Int32Builder).Append(4)

	record := recordBuilder.NewRecord() // this works
	defer record.Release()

	log.Printf("record: %v", record)
	/* prints
2025/05/08 20:32:47 record: record:
schema:
  fields: 1
    - a: type=int32
  rows: 4
  col[0][a]: [1 2 (null) 4]
  */
}

This is not allowed in the Rust implementation:

use std::sync::Arc;

use arrow_array::{Int32Array, RecordBatch};
use arrow_schema::{DataType, Field, Schema};

fn main() {
    let schema = Arc::new(Schema::new(vec![Field::new("a", DataType::Int32, false)]));
    let mut ids = Int32Array:: builder(4);
    ids.append_value(1);
    ids.append_value(2);
    ids.append_null();
    ids.append_value(4);
    let ids = ids.finish();
    // this panics:
    // called `Result::unwrap()` on an `Err` value: InvalidArgumentError("Column 'a' is declared as non-nullable but contains null values")
    let batch = RecordBatch::try_new(schema.clone(), vec![Arc::new(ids)]).unwrap(); 
    println!("batch: {:?}", batch);
}
Component(s)

Other

Dominant language
Assembly
Stars
406
Forks
146
Avg merge
2d 8h
Merged PRs (30d)
93

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/arrow-go

All issues in apache/arrow-go

Similar issues

More Data Engineering issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.