Integration with external logs storage when displaying traces

Many organizations have log aggregation setups, such as shipping all logs to Elasticsearch. For logs corresponding to traced transactions, it may not be feasible or desired to direct them all to the tracing backend. As long as the logs are labeled with trace/span IDs, it is possible to reassemble the logs from the logging backend when viewing traces in Jaeger UI.

This ticket is to track the design and implementation details of such solution.

The high level design is something like this:

+--------------+        +--------------+         +--------------+                    
|              |        |              |         |              |                    
|  Jaeger UI   |------->| jaeger-query |-------->|  logs query  |                    
|              |        |              |         |              |                    
+--------------+        +--------------+         +--------------+                    
                                                        |                            
                                     +------------------+--------------------+       
                                     |                  |                    |       
                             +--------------+    +--------------+    +--------------+
                             | logs storage |    | logs storage |    | logs storage |
                             | Elasticsearch     |   Kafka      |    |    HDFS      |
                             |              |    |              |    |              |
                             +--------------+    +--------------+    +--------------+

The log query service is meant to serve as an abstraction of different log storage types.

Open questions

• what is the API of the log-query service
• does jaeger-query retrieve logs when returning the trace to the UI or on-demand per-span