Background logger prints flush retries straight to the console, with no way to route or rate-limit them

Start at HTTPBackgroundLogger in dist/index.js, especially submitLogsRequest, unwrapLazyValues, registerDroppedItemCount, logFailedPayloadsDir, and dumpDroppedEvents, then trace how initLogger constructs it. Done means the listed retry and dropped-event messages can be routed through the requested injectable logger while preserving console as the default; verify the existing onFlushError and environment options remain unaffected.

Retry / flush failures in HTTPBackgroundLogger dump unstructured console spam and page us

Setup

We run law-server on Kubernetes: long-lived API + Temporal workers, plus 20–30 short-lived agent Job pods at a time. Every pod calls initLogger on start and flushes on exit. Node 22, [email protected]. Logging is off the request path, so a failed ingest write is never user-facing for us.

What happened

Aug 31–Sep 5 we hit nine bursts of /logs3 500s (support ticket 21586 — two separate issues on your side, both since fixed). Biggest one was ~3k failed attempts in 10 minutes.

The dropped spans were fine. The log volume was not. Every pod printed its whole retry chain to stderr as unstructured text. Our alerting is log-based, so each burst paged on-call.

Why we can’t just fix this ourselves

In HTTPBackgroundLogger the failure path writes straight to the console:

• submitLogsRequest — console.warn(errMsg) on each failed attempt, console.info("Sleeping for Ns") between attempts, then console.warn("log request failed after N retries. Dropping batch") (dist/index.js ~L6011–6030)
• unwrapLazyValues retry loop — console.warn(errmsg), console.warn(e), console.info(...) (~L5954)
• registerDroppedItemCount, logFailedPayloadsDir, dumpDroppedEvents — console.warn / console.error (~L6034, L6066, L6118)

Default numTries = 3 → ~5 console lines per failing batch per pod, with payload size and elapsed time interpolated into the message, so there’s no structured field to filter on. 20–30 pods retrying a saturated endpoint = the 3k-line burst.

Everything around this is configurable. The logging isn’t: BRAINTRUST_NUM_RETRIES, BRAINTRUST_MAX_REQUEST_SIZE, BRAINTRUST_QUEUE_DROP_LOGGING_PERIOD, BRAINTRUST_FAILED_PUBLISH_PAYLOADS_DIR. onFlushError is the closest hook, but it doesn’t cover this — it’s only wired through the login params that construct the background logger, and it fires once on the final flush error, after every per-attempt line has already been printed. That leaves monkey-patching console.warn, which we’d rather not ship.

What would help

Either of these works:

1. An injectable sink: initLogger({ logger }), or a global setBraintrustLogger({ debug, info, warn, error }), defaulting to console. We’d point it at our structured logger and emit these at debug with pod/request id attached.
2. Rate-limit the retry lines the same way you already rate-limit the dropped-item counter: one line per failure kind per N seconds, env-configurable. Also have per-attempt errors reach onFlushError (or a sibling callback) so the host app can count failures without parsing strings.

We’d use option 1.

Same shape in the Python SDK

braintrust==0.6.0, _HTTPBackgroundLogger: self.outfile = sys.stderr (logger.py:980). The flush path uses print(..., file=self.outfile) and traceback.print_exc(file=self.outfile) (L1232, L1369–1375, L1392, L1421). That object already has self.logger = logging.getLogger("braintrust") (L1032) but only uses it for one debug line. Routing those prints through that logger would be enough for us — stdlib logging config can take it from there. Happy to open a matching issue on braintrust-sdk-python if you’d rather track it separately.

Context

Ticket 21586, thread with Evan Keith — he suggested filing this here. Following his other rec we now pass projectId alongside projectName at every init site, which killed the project-register calls on pod start. That’s solved and separate from this.

Timestamps and trace IDs

All times UTC, 2026. Pulled from our log store — these are the InternalTraceId values your API returned to the SDK. Two namespaces: law-server (JS SDK) and law-pipelines (Python SDK).

Counts with a + hit our query’s 200-line cap, so they’re floors, not totals.

Window	JS SDK lines	Python SDK lines
Aug 31 19:19–19:53	200+	111
Sep 1 15:19–15:56	21	8
Sep 1 16:42–17:00	4	0
Sep 1 18:01–21:09	66	16
Sep 4 14:41–14:53	2	0
Sep 4 23:50 – Sep 5 01:11	200+	200+

Three other windows we flagged at the time (Aug 31 18:12–18:29, Aug 31 20:17–20:39, Sep 3 17:37–17:54) have no matching lines left in these namespaces, so I left them out.

The error line the SDK prints looks like this:

Error: 500 (Internal Server Error): {"Code":"InternalServerError","InternalTraceId":"c2cc1334d06819049ec0eb183dc622e6","Path":"/logs3","Service":"api"}
log request failed. Elapsed time: 5.077 seconds. Payload size: 3566.

Trace IDs by window:

Sep 1 15:19–15:56

c2cc1334d06819049ec0eb183dc622e6
fbda42fa048688bc006e7bfe04f32400
8b1287c886e03854baf4776e8e636f64
2066defe6c4c5dac3ca1bae19a7e9195
ce353438d1b5e4a097fedf10ce10f910
6a96ef05000000003a6671d73997e54a
6a96ef7d000000002412292dc31ba80a
6a96ef7a00000000614fdcef848738ea
6a96f159000000004919ac781bb38628

Sep 1 16:42–17:00

6a97005b0000000044c223fc902d06ab
6a97005b000000001f4f119eff437961

Sep 1 18:01–21:09

6a9719ac0000000063ca30a5bf5db3dd
3711f5af2f17314aeb6511e6ae9baed5
20d15bc207422036930eb7694fc723b7
c758588ce0d9d7ce9e88a972ae19e9a0
8fc3af82e8c32722145cd748672ae337
6a971627000000001442a38cf7d1a189
6a9716ae000000004ad33535548d0626
6a971771000000007cdc27af335d1c4c
6a9720b3000000005e3235c709ee1b60
6a9732a30000000062330d8e234cd634

Sep 4 23:50 – Sep 5 01:11 (99+ distinct on JS, 96+ on Python)

6a9b5b7f00000000778340d959e03db3
6a9b5b80000000004aa757fac4c721fa
6a9b5b82000000001bbb0dbd2bfee6e2
6a9b5b86000000006701ca1eecf19798
c457ad77f132a16a0a8cf67f2082bfb6
6a9b5b56000000003875e99a0f8f3fb2
6a9b5b57000000006487ced2db512ca5
6a9b5b5a0000000011a1701214f4c907
6a9b5b5e0000000033806e4e6819b65b
6a9b5b660000000057a35ad813a90f56
6a9b5d9b000000004da37a8ec2ee9d70

A different failure shape showed up twice — Sep 1 15:19 (Python SDK) and Sep 4 14:41 (JS SDK, trace id 806e65102cddf74929d6110a3b3e267c): a 400 Failed to decode API key, with connect ETIMEDOUT to a host on port 6543 in the message. One Braintrust request id from the Sep 1 occurrence, if it helps: iad1::rzjs9-1788276485217-bc3e8c663175. Same API key worked immediately before and after both windows.

Can pull the full ID list for any window if that’s useful.