Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

decide() drops billed usage when a call fails after the provider responded

Fermée
  • Quelque chose ne va pas ?
#1,741 0 commentaires 0 réactions 1 personne assignée Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

@tombeckenham y travaille déjà.

Depuis le 13/10/2026.

  • #1742 par @ktrz — ouverte

Évaluation

Cette issue n'a pas encore été évaluée.

Description

has-pr waiting-on: maintainer
TanStack AI version

@tanstack/ai 0.68.0, @tanstack/ai-openai 0.29.1 (current main)

Framework/Library version

None. decide() runs on the server, Node.js v24.14.1.

Describe the bug and the steps to reproduce it

decide() only reports token usage on the success path. When the provider answers and bills the call, but decide() then throws, onUsage never runs and no evaluate:usage event fires. The tokens are billed but never metered. We use decide() in a planner and meter tokens through onUsage, so every one of these failures under-reports our spend.

Two concrete cases on main.

Case 1: the answers cannot be mapped. Shipped adapters pass provider answers through unchecked (for example packages/ai-typesafe/src/adapters/evaluate.ts returns json.answers as is). If an answer is missing, or has the wrong type for its question, mapAnswers throws inside decide() after adapter.evaluate() resolved. result.usage is in scope at that point and is dropped. onError runs, onUsage does not.

Case 2: OpenAI refuses a question. packages/ai-openai/src/adapters/evaluate.ts throws a plain Error('OpenAI declined to answer question "<name>"') from inside the decision.answers.map(...). The response is already in hand, so three things are lost at once: the usage from decision.usage, every other answer in the response, and any typed way to detect a refusal. The caller gets a plain Error with no code, and the adapter log entry has code: undefined. I have no deterministic repro for this one. It needs a real refusal from gpt-6-luna. The code path is visible in the file.

Maintainers already treat lost usage as a bug on the chat side: #1609 (merged) keeps Anthropic usage when a stream stops at max_tokens, and #1735 (open) reports Anthropic usage lost on a stream failure. decide() has the same hole.

Steps to reproduce case 1 on clean main:

  1. Save the test below as packages/ai/tests/decide-usage-repro.test.ts.
  2. Run pnpm nx run @tanstack/ai:test:lib --skip-nx-cache -- tests/decide-usage-repro.test.ts.
  3. The test fails with AssertionError: expected [] to deeply equal [ { promptTokens: 40, …(2) } ]. decide() threw as expected, but hookUsage and eventUsage are both empty.
import { expect, it } from 'vitest'
import { aiEventClient } from '@tanstack/ai-event-client'
import { boolean, decide } from '../src/index'
import type { TokenUsage } from '../src/types'

it('decide() reports billed usage when the answers cannot be mapped', async () => {
  const usage = { promptTokens: 40, completionTokens: 5, totalTokens: 45 }
  const hookUsage: Array<TokenUsage> = []
  const eventUsage: Array<TokenUsage> = []
  const off = aiEventClient.on('evaluate:usage', (e) => eventUsage.push(e.payload.usage), { withEventTarget: true })

  const call = decide({
    adapter: {
      kind: 'evaluate', name: 'repro', model: 'repro-model', '~types': { providerOptions: {} },
      // The provider answered and billed tokens, but the answer has the wrong type.
      evaluate: async () => ({
        model: 'repro-model', usage,
        answers: { refund: { type: 'choice', choice: 'x', probabilities: { x: 1 }, confidence: 1 } },
      }),
    },
    state: 'Please refund the extra payment.',
    questions: { refund: boolean({ instructions: 'Is this a refund request?' }) },
    middleware: [{ onUsage: (_ctx, u) => void hookUsage.push(u) }],
    debug: false,
  })

  await expect(call).rejects.toThrow('expected noul answer for "refund"')
  off()
  expect(hookUsage).toEqual([usage])
  expect(eventUsage).toEqual([usage])
})

Expected: onUsage and evaluate:usage get the usage the provider reported, then onError runs and decide() rethrows. onFinish stays off for a failed call.

Actual: nothing reports the usage.

Your Minimal, Reproducible Example - (Sandbox Highly Recommended)

Inline above. It is a Vitest test against this repo, no sandbox needed.

Screenshots or Videos (Optional)

No response

Do you intend to try to help solve this bug with your own PR?

Yes, I am also opening a PR that solves the problem along side this issue

Terms & Code of Conduct
  • I agree to follow this project's Code of Conduct
  • I understand that if my bug cannot be reliable reproduced in a debuggable environment, it will probably not be fixed and this issue may even be closed.
Langage dominant
TypeScript
Étoiles
3.2k
Forks
365
Merge moyen
1 j 8 h
PR mergées (30 j)
261

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de TanStack/ai

Toutes les issues de TanStack/ai

Issues similaires

Plus d'issues TypeScript

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.