@google-cloud/bigtable: a row-key-only read path that skips per-cell decode cost
Maintainer antworten meist innerhalb von 1 Tag
Dieses Issue hat noch niemand übernommen.
Bewertung
- Schwierigkeit
- 5/5
- Geschätzter Aufwand
- Über eine Woche
- Anfängerfreundlichkeit
- 35/100
- Issue-Typ
- Feature
- Klarheit
- Größtenteils klar
- Aktivitätsstatus
- Aktiv
- Tech-Stack
- google-cloud, nodejs, typescript
Rechercherichtung
Start at the Bigtable Table getRows() and createReadStream() entry points, then trace GetRowsOptions through the readRows response handling and existing generated-client integration. Define how keysOnly returns row keys without materializing cell structures, and verify the behavior and performance against the standard path; the payload names no specific test file to run.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Beschreibung
Library Name
@google-cloud/bigtable
A screenshot that you have tested with "Try this API".
Not applicable — this is a client-library implementation request, not a discrepancy between the client and the raw API. The ReadRows RPC already returns the CellChunk data needed (row key, family, qualifier, value); the request is about how @google-cloud/bigtable decodes and materializes that response client-side, which APIs Explorer can't exercise (it's not something visible in the raw wire response, only in the generated client's object graph).
If the form truly requires an image upload and rejects text, you may need to attach a blank/placeholder image or leave a follow-up comment explaining the same thing after submitting.
What would you like to see in the library?
Summary
For read patterns that only need the row key (not any cell value), the current client still pays the full per-cell decode cost: every CellChunk's familyName/qualifier gets unwrapped into a StringValue/BytesValue message, and the result is reassembled into a full, family-then-qualifier-grouped Row object. A row-key-only consumer never touches any of that.
We'd like to propose a keysOnly read mode that skips it.
Motivation
A possible id pattern is embedding a second identifier directly in the row key (e.g. ${id}${linkedId}, or ${id}${reverse-timestamp}${linkedId}), specifically so a range/prefix scan can answer "what is this entity linked to" without reading any column at all. We have several tables like this, and the read pattern is: scan a prefix, collect just the row keys, parse the linked id back out of each key.
Doing this today through getRows()/createReadStream() means paying for a full Row object per row: family map, qualifier map, cell values, all built and then discarded unread. A server-side StripValueTransformerFilter removes the value, but the client still decodes and reassembles everything else. We also checked the existing decode: false option on GetRowsOptions: it only controls whether a cell's value comes back as a Buffer or a string (Mutation.convertFromBytes), applied after the full Row is already built, so it doesn't help here either.
What we measured
We benchmarked three approaches to this exact pattern, same data, same requests:
- Standard
getRows()/createReadStream(), with a server-side filter (family scope +cellsPerColumnLimitFilter(1)+stripValueTransformer), reading onlyrow.idfrom the result. - The same server-side filter, called via the low-level generated client directly (bypassing the
Tablewrapper), still using the standard generated response decoder. - A hand-written response decoder over a raw
grpc-jsconnection to the same RPC, that reads only therowKeyandcommitRowfields of eachCellChunk(protobuf fields 1 and 9) and callsreader.skipType()on everything else, so no per-cell wrapper objects and no groupedRoware ever built.
On one real user's data, run sequentially with no server-side filter, comparing approach 2 against approach 3:
| Rows | Without keysOnly |
With keysOnly |
Improvement |
|---|---|---|---|
| 26,492 | 204.5ms | 146.4ms | -28% |
| 82,865 | 421.1ms | 198.8ms | -53% |
| 125,606 | 1539.4ms | 646.1ms | -58% |
| 756,377 | 4342.6ms | 1722.4ms | -60% |
The saving grows with row count, which is what we'd expect from removing a fixed per-cell decode cost: the more rows in the scan, the more per-cell wrapper-object construction there was to eliminate.
Across those same tables, approach 3 cut wall-clock time by 53-60% versus approach 2, and turned a workload that was clearly slower than a comparable Java client (using its own standard, fully-materializing read path) into one that matched or beat it — by 1.6x to 3.8x on the tables with the widest rows.
At 1,000 simulated users (capped at 100k rows on the largest table, matching a real production limit), approach 3 averaged 32.0ms per user vs. 42.0ms for the Java comparison, and beat it on p50/p90/p99 as well.
We also tried the newer executeQuery/GoogleSQL SELECT _key FROM ... path as an alternative, official way to get server-side projection down to just the key. On the same 1,000-user benchmark it was 4-8x slower than approach 3 on every percentile, so it isn't a substitute for this request.
Proposed API
Something in the shape of:
js table.createReadStream({ prefix: '...', keysOnly: true }) // or table.getRows({ prefix: '...', keysOnly: true })
returning row keys only (as Buffer/string), without constructing family/qualifier/value structures at all. Internally this could reuse the existing gapic readRows call and swap in a lighter response transform, without touching the generated protobuf layer.
Describe alternatives you've considered
StripValueTransformerFilterremoves the value but not the family/qualifier decode or theRowreassembly cost.decode: falseonGetRowsOptionsonly changes the cell value's return type (Buffervs string), applied after the fullRowis already built.executeQuery(see above) is much slower for this pattern in our testing, likely because it runs a different, newer query-execution path server-side rather than the long-establishedReadRowspath.- A private raw-
grpc-jsdecoder, which is what we're using internally today, works, but means reimplementing retry/credential/channel-pooling logic thatgoogle-gaxalready provides, and isn't something we think every consumer of this client should have to build themselves.
Additional context/notes
Happy to contribute a PR if this is something the maintainers would want, or to share more of our benchmark methodology if useful.
- Vorherrschende Sprache
- TypeScript
- Sterne
- 3.2k
- Forks
- 732
- Ø Merge
- 2 T. 11 Std.
- Gemergte PRs (30 T.)
- 99
Entwicklungsumgebung
- Kein Dockerfile und keine Docker-Compose-Datei
- Hat eine Pull-Request-Vorlage
- Beitragsleitfaden lesen
Erste Schritte
- Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
- Forken Sie das Repository und arbeiten Sie in einem Branch.
- Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.
Mehr aus googleapis/google-cloud-node
-
Broken deprecation chainOffen
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 66/100
googleapis/google-cloud-node#9854 ·
Maintainer antworten meist innerhalb von 1 Tag
-
pubsub: "Operation attempted on ended Span" when a modack is flushed after the message was ackedOffenapi: pubsub
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 82/100
googleapis/google-cloud-node#9853 ·
Maintainer antworten meist innerhalb von 1 Tag
-
storage: V4 getSignedUrl sorts canonical headers with localeCompare, so they disagree with X-Goog-SignedHeadersEvtl. vergeben @itsrayeedscodespace hat das vor 9 Tagen übernommen. Offenapi: storage
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 82/100
googleapis/google-cloud-node#9840 · 1 Kommentar ·
Maintainer antworten meist innerhalb von 1 Tag
-
firestore: getPartitions() emits gax AutopaginateTrueWarning in every processEvtl. vergeben @TheRealDanii hat das vor 13 Tagen übernommen. Offenapi: firestore
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 76/100
googleapis/google-cloud-node#9736 ·
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 1/5 Unter einer Stunde Anfängerfreundlichkeit 82/100
googleapis/google-cloud-node#9723 ·
Maintainer antworten meist innerhalb von 1 Tag
Alle Issues in googleapis/google-cloud-node
Ähnliche Issues
-
area: batch
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 75/100
inabagumi/shinju-date#7294 ·
Maintainer antworten meist innerhalb von 1 Tag
-
documentation
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 68/100
streaming-video-technology-alliance/common-media-library#700 ·
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 82/100
vercel-labs/ui#809 ·
Maintainer antworten meist innerhalb von 1 Tag
-
enhancement
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 67/100
KrisCris/village-in-the-shade-guide#75 ·
Maintainer antworten meist innerhalb von 1 Tag
-
bot:ai-assisted component:platform-js status:untriaged
Schwierigkeit 1/5 Unter einer Stunde Anfängerfreundlichkeit 74/100
midnightntwrk/platform-js#38 ·