CTable Object Specification

This document outlines the specifications for a new CTable object. A CTable will provide a high-performance, disk-based, columnar data structure.

Core Properties

The CTable object will have the following properties:

• Schema-driven: A schema must be defined at creation time.

  • The schema will map column names to their data types (dtypes).
  • It will support all NumPy dtypes except for object.

• Columnar Storage: Data will be stored column by column.

  • Each column will be a separate blosc2.NDArray object, allowing for efficient queries and compression.

• Disk-based Persistence: The CTable will be backed by on-disk storage.

  • A blosc2.TreeStore will be used as the container for the columnar blosc2.NDArray objects.

• Appendable: New data can be efficiently appended to an existing CTable.

  • An append() method will be provided for this purpose.

• Chunked: Data within each column will be stored in chunks.

  • The chunk size will be configurable to optimize for different access patterns and hardware. A sensible default will be provided.

• Write Buffering: Appended data will be temporarily held in an in-memory buffer for performance.

  • The buffer will be a dictionary of NumPy arrays.
  • Data is flushed from the buffer to the persistent disk storage when the buffer is full, when explicitly requested, or when the CTable is closed.

• Context Manager Support: The CTable can be used as a context manager.

  • When exiting the with block, any pending data in the write buffer will be automatically flushed to disk, ensuring data integrity.

• Iterable: The CTable will be iterable.

  • Iteration will yield rows, with each row represented as a dictionary mapping column names to values.

A first attempt is the ColumnTable class in https://github.com/Blosc/python-blosc2/compare/ctable?expand=1#diff-3b44cef6cb47ef2b49256ed85ec414b28540c04dc245220327a3f27f090e4168 . This can be taken as a model for the more complete CTable that should be implemented in src/blosc2/ctable.py. A test suite should be implemented in a new tests/test_ctable.py module.