@google-cloud/bigtable: a row-key-only read path that skips per-cell decode cost
メンテナーはふだん 1 日以内に返信
まだ誰も着手していません。
評価
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 初心者へのやさしさ
- 35/100
- issue の種類
- 機能追加
- 明瞭さ
- おおむね明確
- 活発さ
- 活発
- 技術スタック
- google-cloud, nodejs, typescript
調査の方向性
Start at the Bigtable Table getRows() and createReadStream() entry points, then trace GetRowsOptions through the readRows response handling and existing generated-client integration. Define how keysOnly returns row keys without materializing cell structures, and verify the behavior and performance against the standard path; the payload names no specific test file to run.
索引モデルが issue の本文から書いたものです。
説明
Library Name
@google-cloud/bigtable
A screenshot that you have tested with "Try this API".
Not applicable — this is a client-library implementation request, not a discrepancy between the client and the raw API. The ReadRows RPC already returns the CellChunk data needed (row key, family, qualifier, value); the request is about how @google-cloud/bigtable decodes and materializes that response client-side, which APIs Explorer can't exercise (it's not something visible in the raw wire response, only in the generated client's object graph).
If the form truly requires an image upload and rejects text, you may need to attach a blank/placeholder image or leave a follow-up comment explaining the same thing after submitting.
What would you like to see in the library?
Summary
For read patterns that only need the row key (not any cell value), the current client still pays the full per-cell decode cost: every CellChunk's familyName/qualifier gets unwrapped into a StringValue/BytesValue message, and the result is reassembled into a full, family-then-qualifier-grouped Row object. A row-key-only consumer never touches any of that.
We'd like to propose a keysOnly read mode that skips it.
Motivation
A possible id pattern is embedding a second identifier directly in the row key (e.g. ${id}${linkedId}, or ${id}${reverse-timestamp}${linkedId}), specifically so a range/prefix scan can answer "what is this entity linked to" without reading any column at all. We have several tables like this, and the read pattern is: scan a prefix, collect just the row keys, parse the linked id back out of each key.
Doing this today through getRows()/createReadStream() means paying for a full Row object per row: family map, qualifier map, cell values, all built and then discarded unread. A server-side StripValueTransformerFilter removes the value, but the client still decodes and reassembles everything else. We also checked the existing decode: false option on GetRowsOptions: it only controls whether a cell's value comes back as a Buffer or a string (Mutation.convertFromBytes), applied after the full Row is already built, so it doesn't help here either.
What we measured
We benchmarked three approaches to this exact pattern, same data, same requests:
- Standard
getRows()/createReadStream(), with a server-side filter (family scope +cellsPerColumnLimitFilter(1)+stripValueTransformer), reading onlyrow.idfrom the result. - The same server-side filter, called via the low-level generated client directly (bypassing the
Tablewrapper), still using the standard generated response decoder. - A hand-written response decoder over a raw
grpc-jsconnection to the same RPC, that reads only therowKeyandcommitRowfields of eachCellChunk(protobuf fields 1 and 9) and callsreader.skipType()on everything else, so no per-cell wrapper objects and no groupedRoware ever built.
On one real user's data, run sequentially with no server-side filter, comparing approach 2 against approach 3:
| Rows | Without keysOnly |
With keysOnly |
Improvement |
|---|---|---|---|
| 26,492 | 204.5ms | 146.4ms | -28% |
| 82,865 | 421.1ms | 198.8ms | -53% |
| 125,606 | 1539.4ms | 646.1ms | -58% |
| 756,377 | 4342.6ms | 1722.4ms | -60% |
The saving grows with row count, which is what we'd expect from removing a fixed per-cell decode cost: the more rows in the scan, the more per-cell wrapper-object construction there was to eliminate.
Across those same tables, approach 3 cut wall-clock time by 53-60% versus approach 2, and turned a workload that was clearly slower than a comparable Java client (using its own standard, fully-materializing read path) into one that matched or beat it — by 1.6x to 3.8x on the tables with the widest rows.
At 1,000 simulated users (capped at 100k rows on the largest table, matching a real production limit), approach 3 averaged 32.0ms per user vs. 42.0ms for the Java comparison, and beat it on p50/p90/p99 as well.
We also tried the newer executeQuery/GoogleSQL SELECT _key FROM ... path as an alternative, official way to get server-side projection down to just the key. On the same 1,000-user benchmark it was 4-8x slower than approach 3 on every percentile, so it isn't a substitute for this request.
Proposed API
Something in the shape of:
js table.createReadStream({ prefix: '...', keysOnly: true }) // or table.getRows({ prefix: '...', keysOnly: true })
returning row keys only (as Buffer/string), without constructing family/qualifier/value structures at all. Internally this could reuse the existing gapic readRows call and swap in a lighter response transform, without touching the generated protobuf layer.
Describe alternatives you've considered
StripValueTransformerFilterremoves the value but not the family/qualifier decode or theRowreassembly cost.decode: falseonGetRowsOptionsonly changes the cell value's return type (Buffervs string), applied after the fullRowis already built.executeQuery(see above) is much slower for this pattern in our testing, likely because it runs a different, newer query-execution path server-side rather than the long-establishedReadRowspath.- A private raw-
grpc-jsdecoder, which is what we're using internally today, works, but means reimplementing retry/credential/channel-pooling logic thatgoogle-gaxalready provides, and isn't something we think every consumer of this client should have to build themselves.
Additional context/notes
Happy to contribute a PR if this is something the maintainers would want, or to share more of our benchmark methodology if useful.
- 主要言語
- TypeScript
- スター
- 3.2k
- フォーク
- 732
- 平均マージ
- 2日 11時間
- マージ済み PR(30日)
- 99
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートあり
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
googleapis/google-cloud-node のほかの issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 66/100
googleapis/google-cloud-node#9854 ·
メンテナーはふだん 1 日以内に返信
-
pubsub: "Operation attempted on ended Span" when a modack is flushed after the message was ackedオープンapi: pubsub
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
googleapis/google-cloud-node#9853 ·
メンテナーはふだん 1 日以内に返信
-
storage: V4 getSignedUrl sorts canonical headers with localeCompare, so they disagree with X-Goog-SignedHeaders対応中かも @itsrayeedscodespace が 9 日前に担当しました。 オープンapi: storage
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
googleapis/google-cloud-node#9840 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
firestore: getPartitions() emits gax AutopaginateTrueWarning in every process対応中かも @TheRealDanii が 13 日前に担当しました。 オープンapi: firestore
難易度 2/5 1〜3時間 初心者へのやさしさ 76/100
googleapis/google-cloud-node#9736 ·
メンテナーはふだん 1 日以内に返信
-
難易度 1/5 1時間未満 初心者へのやさしさ 82/100
googleapis/google-cloud-node#9723 ·
メンテナーはふだん 1 日以内に返信
googleapis/google-cloud-node の issue をすべて見る
似ている issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 67/100
ilan-pinto/roaring-lions#1552 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
メンテナーはふだん 1 日以内に返信
-
bug: [React Native/Android] KeepAliveService never starts when CAMERA is not granted (Android 14+)オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
cloudflare/realtimekit-ui#208 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
usebruno/bruno-vscode#235 ·
メンテナーはふだん 3 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
メンテナーはふだん 1 日以内に返信