@google-cloud/bigtable: a row-key-only read path that skips per-cell decode cost
Les mainteneurs répondent en général sous 1 jour
Personne n'a encore pris cette issue.
Évaluation
- Difficulté
- 5/5
- Temps estimé
- Plus d'une semaine
- Accessibilité débutants
- 35/100
- Type d'issue
- Fonctionnalité
- Clarté
- Plutôt claire
- Activité
- Active
- Stack technique
- google-cloud, nodejs, typescript
Piste de recherche
Start at the Bigtable Table getRows() and createReadStream() entry points, then trace GetRowsOptions through the readRows response handling and existing generated-client integration. Define how keysOnly returns row keys without materializing cell structures, and verify the behavior and performance against the standard path; the payload names no specific test file to run.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Description
Library Name
@google-cloud/bigtable
A screenshot that you have tested with "Try this API".
Not applicable — this is a client-library implementation request, not a discrepancy between the client and the raw API. The ReadRows RPC already returns the CellChunk data needed (row key, family, qualifier, value); the request is about how @google-cloud/bigtable decodes and materializes that response client-side, which APIs Explorer can't exercise (it's not something visible in the raw wire response, only in the generated client's object graph).
If the form truly requires an image upload and rejects text, you may need to attach a blank/placeholder image or leave a follow-up comment explaining the same thing after submitting.
What would you like to see in the library?
Summary
For read patterns that only need the row key (not any cell value), the current client still pays the full per-cell decode cost: every CellChunk's familyName/qualifier gets unwrapped into a StringValue/BytesValue message, and the result is reassembled into a full, family-then-qualifier-grouped Row object. A row-key-only consumer never touches any of that.
We'd like to propose a keysOnly read mode that skips it.
Motivation
A possible id pattern is embedding a second identifier directly in the row key (e.g. ${id}${linkedId}, or ${id}${reverse-timestamp}${linkedId}), specifically so a range/prefix scan can answer "what is this entity linked to" without reading any column at all. We have several tables like this, and the read pattern is: scan a prefix, collect just the row keys, parse the linked id back out of each key.
Doing this today through getRows()/createReadStream() means paying for a full Row object per row: family map, qualifier map, cell values, all built and then discarded unread. A server-side StripValueTransformerFilter removes the value, but the client still decodes and reassembles everything else. We also checked the existing decode: false option on GetRowsOptions: it only controls whether a cell's value comes back as a Buffer or a string (Mutation.convertFromBytes), applied after the full Row is already built, so it doesn't help here either.
What we measured
We benchmarked three approaches to this exact pattern, same data, same requests:
- Standard
getRows()/createReadStream(), with a server-side filter (family scope +cellsPerColumnLimitFilter(1)+stripValueTransformer), reading onlyrow.idfrom the result. - The same server-side filter, called via the low-level generated client directly (bypassing the
Tablewrapper), still using the standard generated response decoder. - A hand-written response decoder over a raw
grpc-jsconnection to the same RPC, that reads only therowKeyandcommitRowfields of eachCellChunk(protobuf fields 1 and 9) and callsreader.skipType()on everything else, so no per-cell wrapper objects and no groupedRoware ever built.
On one real user's data, run sequentially with no server-side filter, comparing approach 2 against approach 3:
| Rows | Without keysOnly |
With keysOnly |
Improvement |
|---|---|---|---|
| 26,492 | 204.5ms | 146.4ms | -28% |
| 82,865 | 421.1ms | 198.8ms | -53% |
| 125,606 | 1539.4ms | 646.1ms | -58% |
| 756,377 | 4342.6ms | 1722.4ms | -60% |
The saving grows with row count, which is what we'd expect from removing a fixed per-cell decode cost: the more rows in the scan, the more per-cell wrapper-object construction there was to eliminate.
Across those same tables, approach 3 cut wall-clock time by 53-60% versus approach 2, and turned a workload that was clearly slower than a comparable Java client (using its own standard, fully-materializing read path) into one that matched or beat it — by 1.6x to 3.8x on the tables with the widest rows.
At 1,000 simulated users (capped at 100k rows on the largest table, matching a real production limit), approach 3 averaged 32.0ms per user vs. 42.0ms for the Java comparison, and beat it on p50/p90/p99 as well.
We also tried the newer executeQuery/GoogleSQL SELECT _key FROM ... path as an alternative, official way to get server-side projection down to just the key. On the same 1,000-user benchmark it was 4-8x slower than approach 3 on every percentile, so it isn't a substitute for this request.
Proposed API
Something in the shape of:
js table.createReadStream({ prefix: '...', keysOnly: true }) // or table.getRows({ prefix: '...', keysOnly: true })
returning row keys only (as Buffer/string), without constructing family/qualifier/value structures at all. Internally this could reuse the existing gapic readRows call and swap in a lighter response transform, without touching the generated protobuf layer.
Describe alternatives you've considered
StripValueTransformerFilterremoves the value but not the family/qualifier decode or theRowreassembly cost.decode: falseonGetRowsOptionsonly changes the cell value's return type (Buffervs string), applied after the fullRowis already built.executeQuery(see above) is much slower for this pattern in our testing, likely because it runs a different, newer query-execution path server-side rather than the long-establishedReadRowspath.- A private raw-
grpc-jsdecoder, which is what we're using internally today, works, but means reimplementing retry/credential/channel-pooling logic thatgoogle-gaxalready provides, and isn't something we think every consumer of this client should have to build themselves.
Additional context/notes
Happy to contribute a PR if this is something the maintainers would want, or to share more of our benchmark methodology if useful.
- Langage dominant
- TypeScript
- Étoiles
- 3.2k
- Forks
- 732
- Merge moyen
- 2 j 11 h
- PR mergées (30 j)
- 99
Préparer son environnement
- Aucun Dockerfile ni fichier Docker Compose
- Propose un modèle de pull request
- Lire le guide de contribution
Par où commencer
- Lisez l'issue en entier, puis le guide de contribution du projet.
- Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
- Forkez le dépôt et travaillez sur une branche.
- Ouvrez une pull request qui référence le numéro de l'issue.
Autres issues de googleapis/google-cloud-node
-
Broken deprecation chainOuverte
Difficulté 2/5 1-3 heures Accessibilité débutants 66/100
googleapis/google-cloud-node#9854 ·
Les mainteneurs répondent en général sous 1 jour
-
pubsub: "Operation attempted on ended Span" when a modack is flushed after the message was ackedOuverteapi: pubsub
Difficulté 2/5 1-3 heures Accessibilité débutants 82/100
googleapis/google-cloud-node#9853 ·
Les mainteneurs répondent en général sous 1 jour
-
storage: V4 getSignedUrl sorts canonical headers with localeCompare, so they disagree with X-Goog-SignedHeadersPeut-être pris @itsrayeedscodespace l’a pris il y a 9 jours. Ouverteapi: storage
Difficulté 2/5 1-3 heures Accessibilité débutants 82/100
googleapis/google-cloud-node#9840 · 1 commentaire ·
Les mainteneurs répondent en général sous 1 jour
-
firestore: getPartitions() emits gax AutopaginateTrueWarning in every processPeut-être pris @TheRealDanii l’a pris il y a 13 jours. Ouverteapi: firestore
Difficulté 2/5 1-3 heures Accessibilité débutants 76/100
googleapis/google-cloud-node#9736 ·
Les mainteneurs répondent en général sous 1 jour
-
Difficulté 1/5 Moins d'une heure Accessibilité débutants 82/100
googleapis/google-cloud-node#9723 ·
Les mainteneurs répondent en général sous 1 jour
Toutes les issues de googleapis/google-cloud-node
Issues similaires
-
area: batch
Difficulté 2/5 1-3 heures Accessibilité débutants 75/100
inabagumi/shinju-date#7294 ·
Les mainteneurs répondent en général sous 1 jour
-
documentation
Difficulté 2/5 1-3 heures Accessibilité débutants 68/100
streaming-video-technology-alliance/common-media-library#700 ·
Les mainteneurs répondent en général sous 1 jour
-
Difficulté 2/5 1-3 heures Accessibilité débutants 82/100
vercel-labs/ui#809 ·
Les mainteneurs répondent en général sous 1 jour
-
enhancement
Difficulté 2/5 1-3 heures Accessibilité débutants 67/100
KrisCris/village-in-the-shade-guide#75 ·
Les mainteneurs répondent en général sous 1 jour
-
bot:ai-assisted component:platform-js status:untriaged
Difficulté 1/5 Moins d'une heure Accessibilité débutants 74/100
midnightntwrk/platform-js#38 ·