jaegertracing/jaeger

Integration with external logs storage when displaying traces

Open

#649 opened on Jan 13, 2018

View on GitHub
 (21 comments) (39 reactions) (0 assignees)Go (18,974 stars) (2,326 forks)batch import
enhancementhelp wanted

Description

Many organizations have log aggregation setups, such as shipping all logs to Elasticsearch. For logs corresponding to traced transactions, it may not be feasible or desired to direct them all to the tracing backend. As long as the logs are labeled with trace/span IDs, it is possible to reassemble the logs from the logging backend when viewing traces in Jaeger UI.

This ticket is to track the design and implementation details of such solution.

The high level design is something like this:

+--------------+        +--------------+         +--------------+                    
|              |        |              |         |              |                    
|  Jaeger UI   |------->| jaeger-query |-------->|  logs query  |                    
|              |        |              |         |              |                    
+--------------+        +--------------+         +--------------+                    
                                                        |                            
                                     +------------------+--------------------+       
                                     |                  |                    |       
                             +--------------+    +--------------+    +--------------+
                             | logs storage |    | logs storage |    | logs storage |
                             | Elasticsearch     |   Kafka      |    |    HDFS      |
                             |              |    |              |    |              |
                             +--------------+    +--------------+    +--------------+

The log query service is meant to serve as an abstraction of different log storage types.

Open questions

  • what is the API of the log-query service
  • does jaeger-query retrieve logs when returning the trace to the UI or on-demand per-span

Contributor guide

Tech stack
goelasticsearch
Domain
observabilitybackendapi
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
needs investigation
Prerequisites
Knowledge of distributed tracingFamiliarity with Jaeger architecture
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
15
Research direction
The issue outlines a high level design for integrating external log storage with Jaeger UI. Key open questions include the API of a proposed log query service and whether logs should be fetched per span or on demand per trace. To move forward, investigate existing jaeger query service internals (e.g., in ./cmd/query/) and the Jaeger UI code (in ./jaeger ui/) to understand how data is currently fetched. Explore existing log aggregation solutions like Elasticsearch and Kafka to define the log query API. Review comments in the issue for additional context and potential design constraints.