jaegertracing/jaeger

Jaeger is slowly losing operations that are still available via the API

Open

#2,420 opened on Aug 26, 2020

View on GitHub
 (12 comments) (0 reactions) (1 assignee)Go (18,974 stars) (2,326 forks)batch import
buggood first issuestorage/elasticsearch

Description

Describe the bug I have Jaeger hooked up to Elasticsearch and have it running for over a month. Over time my operations count via the jaeger ui went up to 52, however after about a week I noticed it dropped down to 32 and a day later has risen to 46.

To Reproduce Steps to reproduce the behavior:

  1. Have Jaeger hooked up to Elasticsearch for over a month
  2. Create a number of different spans over the course of that month
  3. Wait several days (sorry I can't be more specific)
  4. Observe the # of operations has dropped

Expected behavior Every historical operation that has spans logged against it should be available via the UI

Version (please complete the following information):

  • OS: Linux
  • Jaeger version: 1.17
  • Deployment: Docker

What troubleshooting steps did you try? Checked the Elasticsearch index and all the historical operations are there (and there's 100 of them) Confirmed that the operations aren't present on the response of via /api/services/SERVICENAME/operations

I've confirmed that I can still get the spans via the api via /api/traces?end=1598868060000000&limit=200&lookback=custom&maxDuration&minDuration&operation=WebRequest%3A%2Fquotes%2F19&service=SERVICENAME&start=1588305660000000 even though the operation name isn't in the UI dropdown

Additional context Just in case it's important, I'm using a const sampler at the moment so it's logging everything !

Contributor guide

Tech stack
goelasticsearchdocker
Domain
backendobservability
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
needs investigation
Prerequisites
GoElasticsearchJaeger architecture
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30
Research direction
Investigate the operations query logic in the Jaeger backend, specifically the Elasticsearch storage plugin. Look at files like `plugin/storage/es/spanstore/operation.go` and the API handler for `/api/services/{service}/operations`. The issue suggests a caching or aggregation inconsistency where the UI shows fewer operations than the API returns. Check if there is a TTL or cache invalidation bug. Also review the `jaeger ui` code that fetches operations, but focus on backend data flow.