ForceFlush never flushes the LoggerProvider, so GenAI audit records are silently dropped when the Actor checkpoints

Open Beginner friendly
#2,759 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
78/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go
Domain
observability

Research direction

Start in go/adk/pkg/telemetry/tracing.go and inspect ForceFlush alongside the newLoggerProvider batch processor setup. Check the existing trace-flush path in go/adk/pkg/a2a/server/server.go and the related compiler setting in go/core/internal/translator/kagent/compiler.go. Done means logger records are flushed before checkpointing using the existing timeout, without losing the current trace behavior.

Written by the indexing model from the issue text.

Description

Disclosure: this issue was written by an AI agent (Claude) while verifying kagent's
audit-logging documentation, and reviewed by a human before filing.

Summary

ForceFlush flushes the TracerProvider only. The LoggerProvider's batch processor is never
flushed, so GenAI audit log records can be lost when Substrate checkpoints the Actor as the
response closes. The loss is silent — no error, no retry, and the records are dropped rather
than deferred to the next resume.

For a feature whose audience is security and compliance review, silent loss is a
correctness problem rather than a caveat.

The code

// go/adk/pkg/telemetry/tracing.go:50-61
func ForceFlush(ctx context.Context) {
	type flusher interface{ ForceFlush(context.Context) error }
	fp, ok := otel.GetTracerProvider().(flusher)     // <-- TracerProvider only
	if !ok {
		return
	}
	flushCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), flushTimeout())
	defer cancel()
	if err := fp.ForceFlush(flushCtx); err != nil {
		otel.Handle(err)
	}
}

newLoggerProvider wraps its exporter in sdklog.NewBatchProcessor, so log records sit in
a batch queue with the same hazard the ForceFlush doc comment describes for spans.
grep -rn 'GetLoggerProvider' go returns zero hits in the repository.

Spans have an opt-in pre-response flush and logs have no equivalent:

  • go/core/internal/translator/kagent/compiler.go:75 sets KAGENT_PRE_RESPONSE_TRACE_FLUSH=true
  • go/adk/pkg/a2a/server/server.go:110 honours it
  • grep -rn 'PRE_RESPONSE' go shows no logging counterpart

Observed

On a kind cluster with logging enabled via Harness.spec.env, two identical single-turn
invokes of the same AgentInstance with the same prompt produced 0 GenAI log records and
then 3. A second invoke did not release the first turn's missing records, so they were
dropped rather than deferred.

Later in the same session, with a 3-replica WorkerPool, delivery was reliable across four
consecutive invokes. So the loss is load- and timing-dependent rather than constant, which
is consistent with a batch-interval race.

Expected

ForceFlush also flushes the LoggerProvider, or a KAGENT_PRE_RESPONSE_LOG_FLUSH-style
opt-in exists that matches the trace behaviour, so a record emitted during a turn is
exported before the Actor is checkpointed.

Suggested fix

Cast otel.GetLoggerProvider() alongside the tracer provider in ForceFlush and flush
both, reusing the existing timeout.

Dominant language
Go
Stars
3.8k
Forks
775
Avg merge
1d 11h
Merged PRs (30d)
145

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from kagent-dev/kagent

All issues in kagent-dev/kagent

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.