Improve Query Performance of GetTrace API for ES data source

Requirement - what kind of business use case are you trying to solve?

Improve the query performance of getTrace API in Jaeger, for Elasticsearch storage.

Problem - what in Jaeger blocks you from solving the requirement?

Currently, for Elasticsearch storage, the GetTrace API queries from max-span-age using search_after query. It's actually important or needed to do so because trace data can be spanned across indices and sometimes even days. But this has severe hit in performance, when the data set or retention period is high and especially when we use a hot-warm kind of architecture because of querying every time on warm nodes even for recent trace data.

Proposal - what do you suggest to solve the problem or improve the existing situation?

Have a way to identify the time of first occurrence of an individual trace, and use it in GetTrace API so that at least the recent data could be queried fast and impact can only be seen when querying older Data. One way here is to have a separate index, which has documents, with fields traceID, and the least startTime for that particular traceID. We can use this index (which obviously has less data) to identify the startTime of a trace before querying. Not sure on the way to implement this in a generalised way. Want to know your thoughts on this.

  PS : We use logstash in our pipeline to index data, and having this data through logstash would be simple