@google-cloud/bigtable: a row-key-only read path that skips per-cell decode cost

オープン
#9,363 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
35/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
google-cloud, nodejs, typescript
領域
api, backend, databases

調査の方向性

Start at the Bigtable Table getRows() and createReadStream() entry points, then trace GetRowsOptions through the readRows response handling and existing generated-client integration. Define how keysOnly returns row keys without materializing cell structures, and verify the behavior and performance against the standard path; the payload names no specific test file to run.

索引モデルが issue の本文から書いたものです。

説明

Library Name

@google-cloud/bigtable

A screenshot that you have tested with "Try this API".

Not applicable — this is a client-library implementation request, not a discrepancy between the client and the raw API. The ReadRows RPC already returns the CellChunk data needed (row key, family, qualifier, value); the request is about how @google-cloud/bigtable decodes and materializes that response client-side, which APIs Explorer can't exercise (it's not something visible in the raw wire response, only in the generated client's object graph).
If the form truly requires an image upload and rejects text, you may need to attach a blank/placeholder image or leave a follow-up comment explaining the same thing after submitting.

What would you like to see in the library?

Summary

For read patterns that only need the row key (not any cell value), the current client still pays the full per-cell decode cost: every CellChunk's familyName/qualifier gets unwrapped into a StringValue/BytesValue message, and the result is reassembled into a full, family-then-qualifier-grouped Row object. A row-key-only consumer never touches any of that.
We'd like to propose a keysOnly read mode that skips it.

Motivation

A possible id pattern is embedding a second identifier directly in the row key (e.g. ${id}${linkedId}, or ${id}${reverse-timestamp}${linkedId}), specifically so a range/prefix scan can answer "what is this entity linked to" without reading any column at all. We have several tables like this, and the read pattern is: scan a prefix, collect just the row keys, parse the linked id back out of each key.

Doing this today through getRows()/createReadStream() means paying for a full Row object per row: family map, qualifier map, cell values, all built and then discarded unread. A server-side StripValueTransformerFilter removes the value, but the client still decodes and reassembles everything else. We also checked the existing decode: false option on GetRowsOptions: it only controls whether a cell's value comes back as a Buffer or a string (Mutation.convertFromBytes), applied after the full Row is already built, so it doesn't help here either.

What we measured

We benchmarked three approaches to this exact pattern, same data, same requests:

  1. Standard getRows()/createReadStream(), with a server-side filter (family scope + cellsPerColumnLimitFilter(1) + stripValueTransformer), reading only row.id from the result.
  2. The same server-side filter, called via the low-level generated client directly (bypassing the Table wrapper), still using the standard generated response decoder.
  3. A hand-written response decoder over a raw grpc-js connection to the same RPC, that reads only the rowKey and commitRow fields of each CellChunk (protobuf fields 1 and 9) and calls reader.skipType() on everything else, so no per-cell wrapper objects and no grouped Row are ever built.

On one real user's data, run sequentially with no server-side filter, comparing approach 2 against approach 3:

Rows Without keysOnly With keysOnly Improvement
26,492 204.5ms 146.4ms -28%
82,865 421.1ms 198.8ms -53%
125,606 1539.4ms 646.1ms -58%
756,377 4342.6ms 1722.4ms -60%

The saving grows with row count, which is what we'd expect from removing a fixed per-cell decode cost: the more rows in the scan, the more per-cell wrapper-object construction there was to eliminate.

Across those same tables, approach 3 cut wall-clock time by 53-60% versus approach 2, and turned a workload that was clearly slower than a comparable Java client (using its own standard, fully-materializing read path) into one that matched or beat it — by 1.6x to 3.8x on the tables with the widest rows.

At 1,000 simulated users (capped at 100k rows on the largest table, matching a real production limit), approach 3 averaged 32.0ms per user vs. 42.0ms for the Java comparison, and beat it on p50/p90/p99 as well.

We also tried the newer executeQuery/GoogleSQL SELECT _key FROM ... path as an alternative, official way to get server-side projection down to just the key. On the same 1,000-user benchmark it was 4-8x slower than approach 3 on every percentile, so it isn't a substitute for this request.

Proposed API

Something in the shape of:

​js table.createReadStream({ prefix: '...', keysOnly: true }) // or table.getRows({ prefix: '...', keysOnly: true }) ​

returning row keys only (as Buffer/string), without constructing family/qualifier/value structures at all. Internally this could reuse the existing gapic readRows call and swap in a lighter response transform, without touching the generated protobuf layer.

Describe alternatives you've considered
  • StripValueTransformerFilter removes the value but not the family/qualifier decode or the Row reassembly cost.
  • decode: false on GetRowsOptions only changes the cell value's return type (Buffer vs string), applied after the full Row is already built.
  • executeQuery (see above) is much slower for this pattern in our testing, likely because it runs a different, newer query-execution path server-side rather than the long-established ReadRows path.
  • A private raw-grpc-js decoder, which is what we're using internally today, works, but means reimplementing retry/credential/channel-pooling logic that google-gax already provides, and isn't something we think every consumer of this client should have to build themselves.
Additional context/notes

Happy to contribute a PR if this is something the maintainers would want, or to share more of our benchmark methodology if useful.

主要言語
TypeScript
スター
3.2k
フォーク
732
平均マージ
2日 11時間
マージ済み PR(30日)
99

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

googleapis/google-cloud-node のほかの issue

googleapis/google-cloud-node の issue をすべて見る

似ている issue

TypeScript の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。