jaegertracing/jaeger

[storage] limitations of Cassandra search on LIMIT and complex queries

Open

#166 opened on May 16, 2017

View on GitHub
 (16 comments) (0 reactions) (0 assignees)Go (18,974 stars) (2,326 forks)batch import
area/storagehelp wanted

Description

When querying for traces using serviceName, operationName and a tag with the default LIMIT of 20, some results might be omitted.

This is because of this logic which does the following:

  1. Retrieve all traceIDs matching the operation name
  2. Retrieve all traceIDs matching tags
  3. Intersect 1 & 2

Because Cassandra doesn't guarantee ordering, this could eliminate results.

I propose that we do the following instead (or in addition to what we do now),

  1. Retrieve all traceIds matching tags
  2. Filter by operation name

The reason for retrieving traceIds matching tags first targets the use case when somebody is searching for a jaeger-debug-id or some other tag with low cardinality, guaranteeing them a result when it exists.

Contributor guide

Tech stack
go
Domain
backenddatabaseobservability
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
Go programmingCassandra knowledgeJaeger storage
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40
Research direction
The issue is in plugin/storage/cassandra/spanstore/reader.go lines 260-276. The current logic retrieves trace IDs for operation names and tags separately then intersects them, but Cassandra's lack of ordering can cause omitted results. The proposed fix reverses the order: first retrieve trace IDs matching tags (for low cardinality tags like jaeger debug id) then filter by operation name. The fix would involve modifying the GetTraceIDs function or creating a new query path. Review the existing code and consider the implementation details discussed in the comments.