[bug-bash] Cloud Logging: filtering on ate.actor.uid misses ateapi lifecycle and crash records (labels vs jsonPayload)

Abierto
  • ¿Algo no está bien?
#3,306 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
55/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
go, kubernetes
Área
observability

Línea de trabajo

Start with internal/actorlog/logger.go and ateattr.ActorLogAttrs to trace how actor identity fields reach Cloud Logging, then inspect cmd/atenet/internal/router/ingress/ingress.go and the actor-related component log call sites. Read the Actor-Attributed Component Logs section in docs/observability.md and compare it with the reported filters. Done means actor history is consistently discoverable and the documentation describes the working query; the issue also calls for replacing nested actor fields in router and ateapi records.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area/observability bug-bash kind/bug prio/P2
What happened?

On GKE with Cloud Logging, filtering an actor's logs by ate.actor.uid, as docs/observability.md describes, does not return the control-plane records about that actor. The identity keys are written in two different places, so no single filter finds both kinds of record:

  • Actor output (container stdout/stderr and the worker's synthetic lifecycle lines such as Actor restoring and Actor checkpointed) carries the keys inside the logging.googleapis.com/labels group. Cloud Logging promotes that group to LogEntry.labels, so these records match labels."ate.actor.uid".
  • Component records written with ateattr.ActorLogAttrs carry the same keys flat at the top level, so they land in jsonPayload."ate.actor.uid" and never match a labels. filter. These include ateapi's Actor state changed and Actor crashed, atelet's and ateworker's Restore/Checkpoint timing breakdown, and ateworker's Actor usage sampled.

For one actor taken through create, wake, suspend and resume:

Filter Entries What matched
labels."ate.actor.uid"="<uid>" 175 Actor stdout, runsc output, 6 worker lifecycle lines. No ateapi records
jsonPayload."ate.actor.uid"="<uid>" 22 All 7 Actor state changed (create, resume, suspend), the timing breakdowns, the usage samples

After I deleted the worker pod hosting the actor, the ERROR Actor crashed record could only be found with jsonPayload."ate.actor.uid". The labels. filter returned 0 entries, so the documented actor-centric query cannot show that the actor crashed.

docs/observability.md ("Actor-Attributed Component Logs") says: "Filtering on ate.actor.uid therefore finds a component record and an actor's own output alike." The "Actor-Centric View" example (labels."ate.actor.name"="test") covers only the actor's own output.

Some records about an actor use neither form. They log a nested object instead of the ate.* keys, so no ate.* filter finds them:

  • atenet-router per-request logs (Request, ResumeActor, ResumeActor result, Route ok) log "actor":{"type":"*ateapi.Actor","atespace":"…","name":"…"}, with no uid.
  • ateapi's Reclaiming the node state of an actor released from a worker whose pod is gone logs the same nested actor object, plus actor_uid.
Expected Behavior

One filter returns an actor's full history: its output, lifecycle lines, state transitions and crashes. Either of these would do:

  1. Emit the identity keys in the same place for both kinds of record. For example, on GCE also put them in the label group for component records, as internal/actorlog already does for container output.
  2. Document the combined filter, labels."ate.actor.uid"="<uid>" OR jsonPayload."ate.actor.uid"="<uid>", and correct the sentence above.

In addition, router and ateapi records about an actor should use ateattr.ActorLogAttrs instead of a nested actor object.

Steps to Reproduce
  1. Install on GKE with the default Cloud Logging, then deploy the counter demo: 
    go run ./cmd/ate-setup deploy ate-system
go run ./cmd/ate-setup deploy demo counter
  2. Create an actor, wake it with a request through atenet-router, suspend it, and wake it again: 
    kubectl ate create actor my-counter -a ate-demo-counter --template counter
# send a request with header "ate-target-actor: ate-demo-counter/my-counter"
kubectl ate suspend actor my-counter -a ate-demo-counter
# send another request
  3. Compare the two filters: 
    ACTOR_UID=$(kubectl ate get actor my-counter -a ate-demo-counter -o json | jq -r .metadata.uid)
gcloud logging read "resource.labels.cluster_name=\"$CLUSTER_NAME\" AND labels.\"ate.actor.uid\"=\"$ACTOR_UID\"" \
  --freshness=2h --format='value(jsonPayload.msg,jsonPayload.message)'
gcloud logging read "resource.labels.cluster_name=\"$CLUSTER_NAME\" AND jsonPayload.\"ate.actor.uid\"=\"$ACTOR_UID\"" \
  --freshness=2h --format='value(jsonPayload.msg,jsonPayload."ate.actor.state")'
  4. Optional: delete the actor's worker pod, wait for ACTOR_STATE_CRASHED, and rerun both filters with AND jsonPayload.msg="Actor crashed". Only the jsonPayload. filter finds the record.
Sandbox Runtime

gVisor (runsc). The split comes from how component records and actor output are written, so it should not depend on the runtime.

Agent Substrate Version / Commit SHA

ffed8551 (main as of 2026-10-26)

Kubernetes Version & Environment

GKE 1.36.4, Cloud Logging through the GKE logging agent, default envoy dataplane

Relevant Logs and Diagnostic Output
# worker pod stdout: actor-attributed through the label group (-> LogEntry.labels)
{"logging.googleapis.com/labels":{"ate.actor.name":"my-counter","ate.actor.uid":"7492b215-…","ate.atespace":"ate-demo-counter","ate.template.atespace":"ate-demo-counter","ate.template.name":"counter"},"message":"Actor restoring","span_id":"…","time":"…","trace_flags":"00","trace_id":"…"}

# ate-api-server stdout: flat keys (-> jsonPayload)
{"level":"INFO","msg":"Actor state changed","ate.atespace":"ate-demo-counter","ate.actor.name":"my-counter","ate.actor.uid":"7492b215-…","ate.template.atespace":"ate-demo-counter","ate.template.name":"counter","ate.actor.operation.name":"resume","ate.actor.state":"running",…}
{"level":"ERROR","msg":"Actor crashed","ate.atespace":"ate-demo-counter","ate.actor.name":"my-counter","ate.actor.uid":"7492b215-…",…,"ate.actor.operation.name":"unknown","ate.actor.state":"crashed",…}

# atenet-router: nested object, no ate.* keys and no uid
{"level":"INFO","msg":"ResumeActor result","actor":{"type":"*ateapi.Actor","atespace":"ate-demo-counter","name":"my-counter"},"state":"ACTOR_STATE_RUNNING","workerIP":"…",…}
Additional Context

Found during the observability scenario of the GA bug bash.

Code pointers:

  • internal/actorlog/logger.go: the label group (logging.googleapis.com/labels on GCE)
  • ateattr.ActorLogAttrs: the flat keys
  • cmd/atenet/internal/router/ingress/ingress.go: the nested actor object
  • docs/observability.md: "Actor-Attributed Component Logs" and "Centralized Logging Backends: Actor-Centric View"
Confirmation
  • I have searched existing issues and verified that this is not a duplicate.
  • I have verified that this issue occurs on the latest commit on main.
Lenguaje dominante
Go
Estrellas
6.1k
Forks
676
Merge medio
2 d 41 min
PR fusionados (30 d)
500

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de agent-substrate/substrate

Todos los issues de agent-substrate/substrate

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.