Keep Jev answer confidence for OpenRouter evaluation models in Vercel AI
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 80/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- typescript
- Domain
- observability
Research direction
Start in packages/server-utils/src/integrations/vercel-ai/vercel-ai-dc-subscriber.ts and inspect withProviderConfidence alongside the providerMetadata shapes described in the issue. Verify that OpenRouter evaluation answers preserve confidence and legend in gen_ai.output.messages, matching the existing typesafe-ai behavior.
Written by the indexing model from the issue text.
Description
Vercel AI experimental_evaluate with openrouter.evaluationModel('typesafe/jev-1.13') (@openrouter/ai-sdk-provider 3.1.0) gives an evaluate span, but the answers in gen_ai.output.messages lose confidence and legend.
The OpenRouter provider moves these fields into providerMetadata.openrouter.answers[<question id>] ({ confidence, legend }). Our withProviderConfidence in packages/server-utils/src/integrations/vercel-ai/vercel-ai-dc-subscriber.ts only reads providerMetadata.typesafe.confidence, which is the shape of @ai-sdk/typesafe-ai.
Expected
Evaluate answers from the OpenRouter provider keep confidence (choice, score) and legend (score), the same as answers from @ai-sdk/typesafe-ai.
Notes
- OpenRouter also sends
providerMetadata.openrouter.usage.costandprovider. We could record these too, but they are optional. - Related: Vercel AI
experimental_evaluate(getsentry/sentry-javascript#24694).
- Dominant language
- TypeScript
- Stars
- 8.8k
- Forks
- 1.9k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 588
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from getsentry/sentry-javascript
-
node:http client span never ends when the response is destroyed without an error (res.destroy())OpenBug Node.js Waiting for: Product Owner
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
getsentry/sentry-javascript#25469 · 1 comment ·
Maintainers usually reply within 1 day
-
Core Feature: Typescript Meta: Breaking
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
getsentry/sentry-javascript#25286 · 2 comments ·
Maintainers usually reply within 1 day
-
javascript
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
getsentry/sentry-javascript#24200 · 2 comments ·
Maintainers usually reply within 1 day
-
Cloudflare Workers javascript Tests
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
getsentry/sentry-javascript#24051 · 1 comment ·
Maintainers usually reply within 1 day
-
Elysia events report as `sentry.javascript.bun`Possibly taken @Shubham-Padkonde claimed this 28 days ago. OpenBug Bun javascript
Difficulty 2/5 1-3 hours Newbie friendliness 92/100
getsentry/sentry-javascript#24045 · 1 comment ·
Maintainers usually reply within 1 day
All issues in getsentry/sentry-javascript
Similar issues
-
backend priority:P3 privacy-fix
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
bug desktop v3 windows
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
esengine/DeepSeek-Reasonix#13697 · 1 comment ·
Maintainers usually reply within 1 day
-
area:security documentation effort:S priority:P2
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Maintainers usually reply within 1 day
-
app:web meta:ops
Difficulty 2/5 Half a day Newbie friendliness 78/100
osu-tournament-rating/otr#1113 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100