batch_evaluation fails on self-hosted v4 events_only deployments (uses unavailable v3 read endpoints)

Open
#1,861 1 comment 0 reactions 1 assignee View on GitHub

@hassiebp is already working on this.

Since Sep 5, 2026.

Assessment

This issue has not been assessed yet.

Description

bug feat-evals sdk-python

Environment

  • Langfuse server: self-hosted v4 (docker.io/langfuse/langfuse:4), default write mode events_only
  • Python SDK: 4.14.4 (also verified against 4.15.1 source)

What happened

Running batch evaluation against existing traces fails. The runner fetches items via endpoints that are unavailable on v4 events_only deployments:

  • scope="traces" → client.api.trace.list → GET /api/public/traces
  • scope="observations" → legacy observations_v1 → GET /api/public/observations

Both return:

"This endpoint is not available on deployments running in Langfuse v4 events_only mode."

Root cause

BatchEvaluationRunner._fetch_batch_with_retry (langfuse/batch_evaluation.py, line 1139 in v4.14.4) still calls the v3 read APIs. Per the v3→v4 migration guide, reads should go through GET /api/public/v2/observations (client-side grouping by traceId, root observation for trace-level io).

Expected

batch_evaluation works on events_only deployments, e.g. by fetching via the v2 observations API.

Workaround

We currently run our own thin loop: fetch v2 observations grouped by traceId → evaluate → create_score (score-create ingestion works fine on v4).

Happy to contribute a PR if the v2-based approach sounds right — glad to align on the design first.

Dominant language
Python
Stars
468
Forks
349
Avg merge
12h 16m
Merged PRs (30d)
27

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from langfuse/langfuse-python

All issues in langfuse/langfuse-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.