Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Consider supporting all the different lancedb table column formats.

Open
  • Is something wrong?
#6 0 comments 0 reactions 1 assignee View on GitHub

@ayao227 is already working on this.

Since Sep 28, 2025.

Assessment

This issue has not been assessed yet.

Description

enhancement

Consider serialize all the different column format using this utility and move the list/vector handling in serialize_arrow_value to a separate method?

This code specifically encodes binary data using base64. Below calls it convert_bytes, but can rename it to
serialize_object or serialize_value?

import base64
from datetime import date, datetime, time, timedelta

import numpy as np
import pyarrow as pa


def _convert_temporal(obj):
    """Convert temporal types to string representation."""
    if isinstance(obj, (datetime, date, time)):
        return obj.isoformat()
    if isinstance(obj, timedelta):
        return obj.total_seconds()
    return str(obj)


def _convert_pyarrow_scalar(obj):
    """Convert PyArrow scalar types to JSON-serializable format."""
    if pa.types.is_binary(obj.type):
        return base64.b64encode(obj.as_py()).decode("utf-8")

    if pa.types.is_temporal(obj.type):
        return _convert_temporal(obj.as_py())

    if pa.types.is_list(obj.type) or pa.types.is_map(obj.type):
        return [convert_bytes(item) for item in obj.as_py()]

    if pa.types.is_struct(obj.type):
        return {
            field.name: convert_bytes(obj.field(field.name).as_py())
            for field in obj.type
        }

    if pa.types.is_floating(obj.type):
        return float(obj.as_py())

    return obj.as_py()


def _convert_container(obj):
    """Convert container types (dict, list, tuple) recursively."""
    if isinstance(obj, dict):
        return {key: convert_bytes(value) for key, value in obj.items()}
    if isinstance(obj, (list, tuple)):
        return [convert_bytes(item) for item in obj]
    return obj


def _convert_basic_types(obj):
    """Convert basic Python types to JSON-serializable format."""
    if isinstance(obj, (bytes, pa.BinaryScalar)):
        return base64.b64encode(obj).decode("utf-8")
    if isinstance(obj, (datetime, date, time)):
        return obj.isoformat()
    if isinstance(obj, timedelta):
        return obj.total_seconds()
    if isinstance(obj, np.number):
        return obj.item()
    return obj


def convert_bytes(obj):
    """
    Recursively convert objects to JSON-serializable format.

    Handles:
    - bytes/PyArrow binary: Base64-encoded string
    - datetime types: ISO format string
    - PyArrow types: Python native types
    - nested types: recursive conversion
    """
    # First try basic type conversions
    result = _convert_basic_types(obj)
    if result is not obj:
        return result

    # Then try container types
    result = _convert_container(obj)
    if result is not obj:
        return result

    # Finally try PyArrow scalar types
    if isinstance(obj, pa.Scalar):
        return _convert_pyarrow_scalar(obj)

    return obj
Dominant language
Python
Stars
65
Forks
8
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from lance-format/lance-data-viewer

All issues in lance-format/lance-data-viewer

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.