Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Proposal: supported external reliability scoring and regression-testing pattern for AgentSession

Aperta
  • Qualcosa non va?
#6,707 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
30/100
Tipo di issue
Funzionalità
Chiarezza
Da chiarire
Stato di attività
Attiva
Stack tecnologico
python
Ambito
observability

Direzione di ricerca

Start with the linked adapter notes, sdk/python/livekit_adapter.py and test_livekit_adapter.py in the TurnTrace prototype to understand its event map and tested behavior. Then review the LiveKit Agents public AgentSession event and lifecycle contracts, and ask maintainers which contribution scope they support. Done means agreeing on a narrow, maintainable integration pattern and its expected behavior before proposing an upstream change.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Feature Type

Would make my life easier

Feature Description

We have built a working prototype that turns a LiveKit AgentSession into a
provider-neutral reliability trace for post-call diagnosis and repeatable
regression testing. It captures final transcripts, user and agent state changes,
response latency, tool execution, interruptions, usage, provider errors and
session completion, then sends the observed facts to an external service for
server-side scoring.

The prototype currently calculates four reliability components—turn handling,
transcript integrity, tool reliability and response latency—and uses the same
normalized event model for six deterministic acceptance scenarios and historical
regression monitoring.

This is not a request to add the complete external product to livekit/agents.
Before preparing an upstream PR, I would like guidance on the smallest useful and
maintainable contribution. Would the project be interested in one of these?

  1. A dependency-light example showing the supported way to observe an
    AgentSession, build a bounded post-session reliability payload and flush it
    during shutdown.
  2. Documentation describing which public session events and ordering guarantees
    external reliability integrations can safely depend on.
  3. A small generic exporter/reporter interface that external reliability tools
    can implement without adding a vendor dependency to the core package.
  4. A reporter that connects LiveKit simulations or CI runs to an external
    regression-testing backend.

I would particularly value maintainer feedback on the following technical
questions:

  • Is attaching public event listeners before session.start() an encouraged
    integration pattern, or should post-session integrations prefer
    session.history, session.usage and recorded session data?
  • Which event contracts are intended to be stable for third-party consumers,
    especially tool lifecycle, interruption, transcription-timeout, usage and
    close events?
  • Are event ordering and terminal delivery sufficiently defined for an adapter
    to produce one deterministic report, or should an upstream example document
    deduplication and fallback rules?
  • Should an example default to metadata-only capture and require an explicit
    opt-in before exporting transcript or tool content?
  • Would a LiveKit simulation/CI reporter be more useful than a runtime
    AgentSession example?

The aim is to contribute a small, tested integration pattern that other
observability and reliability tools can reuse—not to introduce a large framework
or a TurnTrace-specific dependency into LiveKit Agents. If the current approach
relies on the wrong event or lifecycle assumptions, I am happy to revise the
prototype and contribute the narrower primitive or example that maintainers
recommend.

A clear indication that this is or is not aligned with the project would be
valuable before we invest further in extracting an upstream patch.

Workarounds / Alternatives

The current prototype attaches an adapter before session.start() and registers
an async shutdown callback. It normalizes public AgentSession events, guards
against duplicate in-process flushes and sends the completed trace through an
idempotent, token-protected ingestion endpoint.

Keeping this entirely in an external repository works, but it leaves several
questions unanswered for other integration authors: which event surface is the
preferred public contract, how to handle attachment timing and fallback events,
and how to export content without creating privacy or redaction surprises.

OpenTelemetry is another possible transport, but the prototype also needs
application-level facts and deterministic acceptance results. I would welcome
guidance on whether this should instead be expressed as an OTLP consumer or a
custom observability endpoint.

Additional Context

Prototype and implementation notes:

Quick adapter check:

git clone https://github.com/niallsemple/turntrace.git
cd turntrace/sdk/python
python -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/python -m unittest -v test_livekit_adapter

Current verification evidence:

  • The adapter test covers event normalization and duplicate-safe shutdown flush.
  • The example imports and runs against LiveKit Agents 1.6.8.
  • A local worker registered successfully with LiveKit Cloud.
  • The container built remotely and the production worker registered as
    turntrace-front-desk in eu-central.
  • The first real microphone-session trace is still pending. If a maintainer
    reproduces a lifecycle or event-contract problem first, I am happy to turn it
    into a failing test and work from that evidence.

This proposal is intended to complement, not duplicate:

  • livekit/agents#6664 / #6666, which focus on a post-call CRM and analytics report plus webhook
    delivery.
  • livekit/agents#5853 / livekit/agents#6026, which focus on exporting session recordings to third-party
    observability systems.

TurnTrace's distinct use case is event-level reliability diagnosis, deterministic
failure scenarios and regression detection across repeated runs. I am happy to
test against the direction chosen for those efforts and adjust the proposed
scope accordingly.

Lingua principale
Python
Stelle
14.6k
Fork
3.9k
Merge medio
2g 23h
PR unite (30g)
177

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di livekit/agents

Tutte le issue di livekit/agents

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.