Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[RFC] Cloud Bigtable Storage Backend

Abierto
  • ¿Algo no está bien?
#185 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
25/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Tranquilo
Stack tecnológico
google-cloud, postgresql, rust
Área
backend, database, distributed-systems

Línea de trabajo

Start by locating the StorageEngine trait and existing storage backend implementations to understand the required operations. Then review how the project configures storage backends and handles transactions, GSIs, and TTL. The proposal describes the expected Bigtable table layout and configuration; completion means implementing the mandatory trait subset with the stated DynamoDB compatibility, transaction isolation, and TTL behavior.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

RFC

Summary

Defines the design and implementation of a production-grade Cloud Bigtable storage backend for ExtendDB. It specifies the mapping of the DynamoDB data model, query engine, transaction controls, and streams onto Google Cloud Bigtable. The design focuses on ensuring that workloads utilizing the DynamoDB API can run on GCP with high fidelity, guaranteeing transaction isolation, serializability, and scalable Time-to-Live (TTL) eviction.

Motivation

ExtendDB's goal is to allow developers to use the DynamoDB programming model in environments where the managed DynamoDB service is unavailable. Implementing a production-ready Cloud Bigtable connector supports this mission by:

  1. Enabling Multi-Cloud Portability
  2. Supporting Hybrid Deployments
  3. Ensuring API Fidelity

Proposed design

Describe the design in enough detail that someone familiar with ExtendDB could implement it. Include:

  • API surface (new operations, changed behavior, wire format)
    • Implements the full mandatory subset of the StorageEngine trait.
    • Enforces strict GSI projection checks: queries requesting non-projected attributes on a GSI return
    • ValidationException (matching DynamoDB behavior) instead of performing slow base-table fetches.
  • Storage implications (schema changes, new tables, migration path)
    • Catalog Metadata: System metadata, accounts, and policies are stored in a single table named __extenddb_catalog__.
    • Data Tables: DynamoDB tables map to Bigtable tables named t<table_id_hex> (where table_id_hex is a 32-character hex UUID). Data attributes are stored in column family d.
    • GSIs: Shadow tables are named t<table_id_hex>_g<idx_hash> (where idx_hash is an 8-character hash of the index name) to respect Bigtable's 50-character limit.
    • Transaction Intents: Column family m in data tables stores locks.
    • TTL Index Table: Introduces a dedicated, sharded shadow table __extenddb_ttl_index__ with key encoding [shard_id:1] [expiry_timestamp_be:8] [account_id_len:1] [account_id] [table_name_len:1] [table_name] [encoded_base_row_key]. This allows targeted range scans for expired items rather than scanning entire data tables.
  • Configuration (new extenddb.toml sections or flags)
    • Adds the following [storage.bigtable] block to extenddb.toml:
[storage]
backend = "bigtable"

[storage.bigtable]
project_id = "my-project"
instance_id = "my-instance"
data_instance_id = "my-data-instance"      # Optional (defaults to instance_id)
credentials_path = "/path/to/sa-key.json" # Optional (falls back to ADC)
pool_size = 20

DynamoDB compatibility

How does this relate to the real DynamoDB API? Is this:

  • Matching existing DynamoDB behavior
  • Extending beyond DynamoDB (ExtendDB-specific)
  • Deliberately diverging from DynamoDB (explain why)

Alternatives considered

  1. Unprotected Single-Row Writes: Bypassing intent locks for non-transactional writes simplifies implementation but permits dirty writes/overwrites on active transactions, breaking serializability. Rejected to preserve API correctness.
  2. Scan-Based TTL Sweeper: Scanning the entire base table to evict expired items is simple to implement but does not scale beyond small tables and creates massive read overhead. Rejected in favor of the sharded TTL index table.
  3. Cloud Spanner Backend: Cloud Spanner natively supports multi-row transactions. This is a very viable alternative, but it has a different performance, indexing, and pricing profile compared to Bigtable. A Spanner connector should be built as a separate, distinct backend.

Breaking changes

None

Open questions

List anything unresolved that needs discussion before implementation.

  1. Tombstone Management on TTL Index: The extenddb_ttl_index table will accumulate empty row tombstones as keys expire. We need to define optimal Bigtable Garbage Collection (GC) policies to ensure compaction cleans them up efficiently.
  2. GSI Reconciliation Frequency: How often should the background GSI reconciler scan for mismatches? It should be configurable to allow operators to balance consistency latency against node load.
Lenguaje dominante
Rust
Estrellas
517
Forks
60
Merge medio
2 d 23 h
PR fusionados (30 d)
25

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de ExtendDB/extenddb

Todos los issues de ExtendDB/extenddb

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.