jaegertracing/jaeger

Jaeger Query can OOM when retrieving large traces

Open

#1051 opened on Sep 5, 2018

View on GitHub
 (38 comments) (1 reaction) (0 assignees)Go (18,974 stars) (2,326 forks)batch import
area/storagebughelp wantedperformance

Description

Requirement - what kind of business use case are you trying to solve?

  • Retrieve large traces
  • Be resilient against bad instrumentation using same traceID for all traces

Problem - what in Jaeger blocks you from solving the requirement?

Jaeger Query OOMs on retrieval of large traces on Cassandra. If someone is crafty, they can easily create a trace with millions of spans, and attempt to retrieve it to systematically bring down all jaeger-query instances.

Proposed Solution - Cassandra

We might do some combination of the following:

  • Trace retrieval limits: Test that the number of spans per trace is less than a user defined threshold before retrieving spans.
  • Protect against large spans submitted on the HTTP POST endpoints by setting a user defined span size limit.
  • Limit number of concurrent requests served by the HTTP GET handler so that we can accurately predict and bound worst case memory utilization.

Any open questions to address

  • Does this affect ES as well?

Contributor guide

Tech stack
goelasticsearch
Domain
backendperformancesecurity
Issue type
performance
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
GoCassandraDistributed tracing conceptsMemory management
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
15
Research direction
Investigate the trace retrieval code in the Jaeger Query service, particularly the span retrieval logic for Cassandra backend. Review the discussion comments (#1051) for proposed solutions like trace retrieval limits and span size limits. Examine existing configuration options and memory management in the query service to implement a configurable threshold for maximum spans per trace.