Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

decide() drops billed usage when a call fails after the provider responded

Cerrado
  • ¿Algo no está bien?
#1,741 0 comentarios 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@tombeckenham ya está trabajando en esto.

Desde el 13/10/2026.

  • #1742 de @ktrz — abierto

Evaluación

Este issue todavía no se ha evaluado.

Descripción

has-pr waiting-on: maintainer
TanStack AI version

@tanstack/ai 0.68.0, @tanstack/ai-openai 0.29.1 (current main)

Framework/Library version

None. decide() runs on the server, Node.js v24.14.1.

Describe the bug and the steps to reproduce it

decide() only reports token usage on the success path. When the provider answers and bills the call, but decide() then throws, onUsage never runs and no evaluate:usage event fires. The tokens are billed but never metered. We use decide() in a planner and meter tokens through onUsage, so every one of these failures under-reports our spend.

Two concrete cases on main.

Case 1: the answers cannot be mapped. Shipped adapters pass provider answers through unchecked (for example packages/ai-typesafe/src/adapters/evaluate.ts returns json.answers as is). If an answer is missing, or has the wrong type for its question, mapAnswers throws inside decide() after adapter.evaluate() resolved. result.usage is in scope at that point and is dropped. onError runs, onUsage does not.

Case 2: OpenAI refuses a question. packages/ai-openai/src/adapters/evaluate.ts throws a plain Error('OpenAI declined to answer question "<name>"') from inside the decision.answers.map(...). The response is already in hand, so three things are lost at once: the usage from decision.usage, every other answer in the response, and any typed way to detect a refusal. The caller gets a plain Error with no code, and the adapter log entry has code: undefined. I have no deterministic repro for this one. It needs a real refusal from gpt-6-luna. The code path is visible in the file.

Maintainers already treat lost usage as a bug on the chat side: #1609 (merged) keeps Anthropic usage when a stream stops at max_tokens, and #1735 (open) reports Anthropic usage lost on a stream failure. decide() has the same hole.

Steps to reproduce case 1 on clean main:

  1. Save the test below as packages/ai/tests/decide-usage-repro.test.ts.
  2. Run pnpm nx run @tanstack/ai:test:lib --skip-nx-cache -- tests/decide-usage-repro.test.ts.
  3. The test fails with AssertionError: expected [] to deeply equal [ { promptTokens: 40, …(2) } ]. decide() threw as expected, but hookUsage and eventUsage are both empty.
import { expect, it } from 'vitest'
import { aiEventClient } from '@tanstack/ai-event-client'
import { boolean, decide } from '../src/index'
import type { TokenUsage } from '../src/types'

it('decide() reports billed usage when the answers cannot be mapped', async () => {
  const usage = { promptTokens: 40, completionTokens: 5, totalTokens: 45 }
  const hookUsage: Array<TokenUsage> = []
  const eventUsage: Array<TokenUsage> = []
  const off = aiEventClient.on('evaluate:usage', (e) => eventUsage.push(e.payload.usage), { withEventTarget: true })

  const call = decide({
    adapter: {
      kind: 'evaluate', name: 'repro', model: 'repro-model', '~types': { providerOptions: {} },
      // The provider answered and billed tokens, but the answer has the wrong type.
      evaluate: async () => ({
        model: 'repro-model', usage,
        answers: { refund: { type: 'choice', choice: 'x', probabilities: { x: 1 }, confidence: 1 } },
      }),
    },
    state: 'Please refund the extra payment.',
    questions: { refund: boolean({ instructions: 'Is this a refund request?' }) },
    middleware: [{ onUsage: (_ctx, u) => void hookUsage.push(u) }],
    debug: false,
  })

  await expect(call).rejects.toThrow('expected noul answer for "refund"')
  off()
  expect(hookUsage).toEqual([usage])
  expect(eventUsage).toEqual([usage])
})

Expected: onUsage and evaluate:usage get the usage the provider reported, then onError runs and decide() rethrows. onFinish stays off for a failed call.

Actual: nothing reports the usage.

Your Minimal, Reproducible Example - (Sandbox Highly Recommended)

Inline above. It is a Vitest test against this repo, no sandbox needed.

Screenshots or Videos (Optional)

No response

Do you intend to try to help solve this bug with your own PR?

Yes, I am also opening a PR that solves the problem along side this issue

Terms & Code of Conduct
  • I agree to follow this project's Code of Conduct
  • I understand that if my bug cannot be reliable reproduced in a debuggable environment, it will probably not be fixed and this issue may even be closed.
Lenguaje dominante
TypeScript
Estrellas
3.2k
Forks
361
Merge medio
1 d 8 h
PR fusionados (30 d)
266

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de TanStack/ai

Todos los issues de TanStack/ai

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.