bigtable: mutateInternal builds self-referential error (err.errors contains err) — breaks pino / AggregateError-aware serializers

Open Beginner friendly
#8,075 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
68/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
nodejs, typescript
Domain
databases

Research direction

Start in handwritten/bigtable/src/utils/mutateInternal.ts and trace the Table#mutate or Table#insert path that handles RPC failures with pending entries. Ensure the resulting error.errors entries are not self-references, then verify that the error remains safe for AggregateError-aware serializers without losing the reported RPC failure details.

Written by the indexing model from the issue text.

Description

api: bigtable
Environment
  • @google-cloud/bigtable version: 6.5.0 (also reproduces on main, see below)
  • Node.js: v24.14.1
  • OS: Linux (Debian 12) — not OS-specific
Summary

When Table#mutate / Table#insert encounters an RPC-level failure with entries still pending and no per-entry mutation errors, mutateInternal sets err.errors to an array that includes err itself, producing a self-referential error object.

This makes the returned error object unsafe to serialize with any tool that follows the AggregateError convention of recursing into err.errors (e.g. pino-std-serializers, most structured loggers, some error-reporting SDKs) — it causes RangeError: Maximum call stack size exceeded.

Offending code

https://github.com/googleapis/google-cloud-node/blob/main/handwritten/bigtable/src/utils/mutateInternal.ts

(Same code also present in the now-archived googleapis/nodejs-bigtable repo at src/utils/mutateInternal.ts.)

if (err) {
  /* If there's an RPC level failure and the mutation entries don't have
     a status code, the RPC level failure error code will be used as the
     entry failure code.
    */
  (err as ServiceError & {errors?: ServiceError[]}).errors =
    mutationErrors.concat(
      [...pendingEntryIndices]
        .filter(index => !mutationErrorsByEntryIndex.has(index))
        .map(() => err),        // ← pushes `err` into its own `err.errors`
    );
  collectMetricsCallback(err, err);
  return;
}

For each pending entry with no per-row status, the outer err is pushed into err.errors. After this runs, err.errors[i] === err for every i, forming a cycle.

Minimal repro
const { pino } = require('pino');
const log = pino();

// Shape of what `mutateInternal` hands back on RPC-level failure with pending entries:
const err = new Error('RPC failure');
err.code = 14; // UNAVAILABLE
err.errors = [err, err, err]; // one self-reference per pending entry

log.error({ err }, 'mutateRows failed');
// RangeError: Maximum call stack size exceeded
//     at errSerializer (pino-std-serializers/lib/err.js:25)
Observed impact

In our production service this has fired thousands of times after a single bigtable RPC failure — each error crashes the logging pipeline before the underlying BigTable failure can be reported, so we lost all upstream diagnostic detail about the original RPC error.

Suggested fix

Don't reuse the outer err as a placeholder for pending entries. Options:

  1. Use a clone of err (a fresh Error with the same message / code / details) for each pending-entry placeholder, so the aggregate contains siblings, not self-references.
  2. Use a distinct sentinel error (e.g. new Error('pending entry — RPC failed with: ' + err.message)) for each pending entry.
  3. Or simply skip padding with the outer error for pending entries and keep err.errors = mutationErrors — the top-level err.message/err.code already communicate the RPC-level failure.

All three avoid the cycle and remain serializable by standard logging / telemetry libraries.

Workaround for current users

Until this is fixed in a release, consumers have to add a cycle-breaking serializer in front of their logger (we did this via a custom pino err serializer that walks err.errors with a WeakSet and replaces cyclic references before delegating to stdSerializers.err).

Dominant language
TypeScript
Stars
3.2k
Forks
714
Avg merge
2d 13h
Merged PRs (30d)
106

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from googleapis/google-cloud-node

All issues in googleapis/google-cloud-node

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.