jaegertracing/jaeger

Jaeger Query can OOM when retrieving large traces

Open

#1,051 opened on 2018年9月5日

GitHub で見る
 (38 comments) (1 reaction) (0 assignees)Go (18,974 stars) (2,326 forks)batch import
area/storagebughelp wantedperformance

説明

Requirement - what kind of business use case are you trying to solve?

  • Retrieve large traces
  • Be resilient against bad instrumentation using same traceID for all traces

Problem - what in Jaeger blocks you from solving the requirement?

Jaeger Query OOMs on retrieval of large traces on Cassandra. If someone is crafty, they can easily create a trace with millions of spans, and attempt to retrieve it to systematically bring down all jaeger-query instances.

Proposed Solution - Cassandra

We might do some combination of the following:

  • Trace retrieval limits: Test that the number of spans per trace is less than a user defined threshold before retrieving spans.
  • Protect against large spans submitted on the HTTP POST endpoints by setting a user defined span size limit.
  • Limit number of concurrent requests served by the HTTP GET handler so that we can accurately predict and bound worst case memory utilization.

Any open questions to address

  • Does this affect ES as well?

コントリビューターガイド

技術スタック
goelasticsearch
領域
backendperformancesecurity
Issue 種別
performance
難度新しい貢献者にとっての実装難度です。1 は非常に小さな変更、5 は専門的な作業です。
4
推定時間経験ある貢献者が調査、実装、テスト、pull request 準備にかけるおおよその時間範囲です。
3-5 days
活動状況Issue が今どれだけ取り組みやすいかを示します。新しい、活発、古い、ブロック中、またはメンテナー入力待ちなどです。
stale
明確さ期待される変更、受け入れ条件、次の手順がどれだけ明確かを示します。
mostly clear
前提条件
GoCassandraDistributed tracing conceptsMemory management
初心者向け度初回貢献者にどれだけ取り組みやすいかを 1-100 で推定したスコアです。
15
調査方針
Investigate the trace retrieval code in the Jaeger Query service, particularly the span retrieval logic for Cassandra backend. Review the discussion comments (#1051) for proposed solutions like trace retrieval limits and span size limits. Examine existing configuration options and memory management in the query service to implement a configurable threshold for maximum spans per trace.
Jaeger Query can OOM when retrieving large traces · jaegertracing/jaeger#1051 | Good First Issue