Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

[RFC] Cloud Bigtable Storage Backend

オープン
  • 何か問題がありますか？
#185 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 2 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
25/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
静か
技術スタック
google-cloud, postgresql, rust
領域
backend, database, distributed-systems

調査の方向性

Start by locating the StorageEngine trait and existing storage backend implementations to understand the required operations. Then review how the project configures storage backends and handles transactions, GSIs, and TTL. The proposal describes the expected Bigtable table layout and configuration; completion means implementing the mandatory trait subset with the stated DynamoDB compatibility, transaction isolation, and TTL behavior.

索引モデルが issue の本文から書いたものです。

説明

RFC

Summary

Defines the design and implementation of a production-grade Cloud Bigtable storage backend for ExtendDB. It specifies the mapping of the DynamoDB data model, query engine, transaction controls, and streams onto Google Cloud Bigtable. The design focuses on ensuring that workloads utilizing the DynamoDB API can run on GCP with high fidelity, guaranteeing transaction isolation, serializability, and scalable Time-to-Live (TTL) eviction.

Motivation

ExtendDB's goal is to allow developers to use the DynamoDB programming model in environments where the managed DynamoDB service is unavailable. Implementing a production-ready Cloud Bigtable connector supports this mission by:

  1. Enabling Multi-Cloud Portability
  2. Supporting Hybrid Deployments
  3. Ensuring API Fidelity

Proposed design

Describe the design in enough detail that someone familiar with ExtendDB could implement it. Include:

  • API surface (new operations, changed behavior, wire format)
    • Implements the full mandatory subset of the StorageEngine trait.
    • Enforces strict GSI projection checks: queries requesting non-projected attributes on a GSI return
    • ValidationException (matching DynamoDB behavior) instead of performing slow base-table fetches.
  • Storage implications (schema changes, new tables, migration path)
    • Catalog Metadata: System metadata, accounts, and policies are stored in a single table named __extenddb_catalog__.
    • Data Tables: DynamoDB tables map to Bigtable tables named t<table_id_hex> (where table_id_hex is a 32-character hex UUID). Data attributes are stored in column family d.
    • GSIs: Shadow tables are named t<table_id_hex>_g<idx_hash> (where idx_hash is an 8-character hash of the index name) to respect Bigtable's 50-character limit.
    • Transaction Intents: Column family m in data tables stores locks.
    • TTL Index Table: Introduces a dedicated, sharded shadow table __extenddb_ttl_index__ with key encoding [shard_id:1] [expiry_timestamp_be:8] [account_id_len:1] [account_id] [table_name_len:1] [table_name] [encoded_base_row_key]. This allows targeted range scans for expired items rather than scanning entire data tables.
  • Configuration (new extenddb.toml sections or flags)
    • Adds the following [storage.bigtable] block to extenddb.toml:
[storage]
backend = "bigtable"

[storage.bigtable]
project_id = "my-project"
instance_id = "my-instance"
data_instance_id = "my-data-instance"      # Optional (defaults to instance_id)
credentials_path = "/path/to/sa-key.json" # Optional (falls back to ADC)
pool_size = 20

DynamoDB compatibility

How does this relate to the real DynamoDB API? Is this:

  • Matching existing DynamoDB behavior
  • Extending beyond DynamoDB (ExtendDB-specific)
  • Deliberately diverging from DynamoDB (explain why)

Alternatives considered

  1. Unprotected Single-Row Writes: Bypassing intent locks for non-transactional writes simplifies implementation but permits dirty writes/overwrites on active transactions, breaking serializability. Rejected to preserve API correctness.
  2. Scan-Based TTL Sweeper: Scanning the entire base table to evict expired items is simple to implement but does not scale beyond small tables and creates massive read overhead. Rejected in favor of the sharded TTL index table.
  3. Cloud Spanner Backend: Cloud Spanner natively supports multi-row transactions. This is a very viable alternative, but it has a different performance, indexing, and pricing profile compared to Bigtable. A Spanner connector should be built as a separate, distinct backend.

Breaking changes

None

Open questions

List anything unresolved that needs discussion before implementation.

  1. Tombstone Management on TTL Index: The extenddb_ttl_index table will accumulate empty row tombstones as keys expire. We need to define optimal Bigtable Garbage Collection (GC) policies to ensure compaction cleans them up efficiently.
  2. GSI Reconciliation Frequency: How often should the background GSI reconciler scan for mismatches? It should be configurable to allow operators to balance consistency latency against node load.
主要言語
Rust
スター
517
フォーク
60
平均マージ
2日 23時間
マージ済み PR（30日）
25

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

ExtendDB/extenddb のほかの issue

ExtendDB/extenddb の issue をすべて見る

似ている issue

Rust の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。