jaegertracing/jaeger

Jaeger Query can OOM when retrieving large traces

Open

#1,051 建立於 2018年9月5日

在 GitHub 查看
 (38 留言) (1 反應) (0 負責人)Go (18,974 star) (2,326 fork)batch import
area/storagebughelp wantedperformance

描述

Requirement - what kind of business use case are you trying to solve?

  • Retrieve large traces
  • Be resilient against bad instrumentation using same traceID for all traces

Problem - what in Jaeger blocks you from solving the requirement?

Jaeger Query OOMs on retrieval of large traces on Cassandra. If someone is crafty, they can easily create a trace with millions of spans, and attempt to retrieve it to systematically bring down all jaeger-query instances.

Proposed Solution - Cassandra

We might do some combination of the following:

  • Trace retrieval limits: Test that the number of spans per trace is less than a user defined threshold before retrieving spans.
  • Protect against large spans submitted on the HTTP POST endpoints by setting a user defined span size limit.
  • Limit number of concurrent requests served by the HTTP GET handler so that we can accurately predict and bound worst case memory utilization.

Any open questions to address

  • Does this affect ES as well?

貢獻者指南

技術棧
goelasticsearch
領域
backendperformancesecurity
議題類型
performance
難度面向新貢獻者的預計實作難度，1 表示很小改動，5 表示專家級工作。
4
預計時間有經驗貢獻者完成調查、實作、測試並準備 pull request 的粗略時間範圍。
3-5 days
活動狀態議題目前的可參與程度：新鮮、活躍、陳舊、阻塞或等待維護者輸入。
stale
清晰度議題是否清楚說明預期改動、驗收標準和下一步。
mostly clear
前置要求
GoCassandraDistributed tracing conceptsMemory management
新手友善度1-100 的估計分數，表示該議題對首次貢獻者的友善程度。
15
研究方向
Investigate the trace retrieval code in the Jaeger Query service, particularly the span retrieval logic for Cassandra backend. Review the discussion comments (#1051) for proposed solutions like trace retrieval limits and span size limits. Examine existing configuration options and memory management in the query service to implement a configurable threshold for maximum spans per trace.