Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Pi Coding Agent instrumentation double-counts tokens and estimated cost in trace rollups (AgentSession.prompt carries summed LLM usage)

Open Beginner friendly
  • Is something wrong?
#2,659 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@sharonadhitya is already working on this.

Since Oct 24, 2026.

  • #2660 by @sharonadhitya — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
typescript
Domain
observability

Research direction

Start by reading js/src/instrumentation/plugins/pi-coding-agent-plugin.ts, especially finishPiLlmSpan and finishPiPromptRun, to see how child usage is added to the prompt span and logged. Trace one prompt with an LLM call and compare its rollup with the leaf LLM metrics. Done when the trace rollup counts the usage and estimated cost once.

Written by the indexing model from the issue text.

Description

Summary

With wrapPiCodingAgentSDK (braintrust 3.39.0, @earendil-works/pi-coding-agent 1.4.0), each AgentSession.prompt task span gets the summed token usage of its child llm spans, and it also carries metadata.model. Braintrust's trace rollups add the parent's metrics to the children's. As a result, the trace summary, the logs table and the trace header report exactly 2× the prompt, completion and cached tokens and 2× estimated_cost for every Pi run.

The instrumentation spec says parent task spans "MAY include aggregate token metrics for display, but SDKs SHOULD avoid making parent rollups look like independent billable LLM calls" (braintrust-spec/skills/instrumentation-spec/references/features/token-and-cost-metrics.md). Here the parent is priced and summed like an LLM call.

Where

js/src/instrumentation/plugins/pi-coding-agent-plugin.ts at [email protected]:

  • finishPiLlmSpan logs the usage on the LLM span and also calls addMetrics(promptState.metrics, usageMetrics) (L703-L707).
  • finishPiPromptRun logs cleanMetrics(state.metrics) on the AgentSession.prompt span, together with extractModelMetadata(...) (L655-L667).

Minimal reproduction

  1. Call braintrust.wrapPiCodingAgentSDK(pi), create an AgentSession with any model, and run one session.prompt(...) that takes at least one LLM turn under initLogger(...).
  2. Run these two project-log queries on that trace:
-- trace rollup
SELECT id, metrics.prompt_tokens, metrics.estimated_cost
FROM project_logs('<project_id>', shape => 'summary') WHERE root_span_id = '<root>'

-- per span, grouped
SELECT span_attributes.name, span_attributes.type, sum(metrics.prompt_tokens) AS pt, sum(metrics.estimated_cost) AS cost
FROM project_logs('<project_id>') WHERE root_span_id = '<root>' AND metrics.prompt_tokens IS NOT NULL
GROUP BY 1, 2

Observed on one trace (gpt-6.1-sol via openai.responses.create, 11 LLM calls):

span type prompt_tokens estimated_cost
openai.responses.create (×11) llm 443,901 0.1708972
AgentSession.prompt task 443,901 0.1708972
summary rollup 887,802 0.3417944

Across 19 consecutive traces, the ratio of summary to leaf-llm sums was 2.000 for both tokens and cost in every trace.

Expected

A trace's rollup equals the sum of its leaf llm spans. Either the prompt span should not carry summed usage (or should carry it under a non-rolled-up name), or it should not be priced or summed as a billable call.

Workaround we use

We sum span_attributes.type = 'llm' AND span_attributes.purpose IS NULL spans and ignore the summary's token and cost metrics.

Dominant language
TypeScript
Stars
29
Forks
21
Avg merge
1d 13h
Merged PRs (30d)
67

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from braintrustdata/braintrust-sdk-javascript

All issues in braintrustdata/braintrust-sdk-javascript

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.