jaegertracing/jaeger

Productionize Streaming Jobs for Service Dependencies

Open

#4,590 opened on Jul 18, 2023

View on GitHub
 (17 comments) (0 reactions) (0 assignees)Go (18,974 stars) (2,326 forks)batch import
enhancementhelp wantedpossible mentorship

Description

Currently we have two analytics solutions for generating service maps:

  • Jaeger Analytics Flink
    • Real time streaming, requires Kafka.
    • More feature rich, includes code for both 1-hop and transitive dependency graphs -- https://www.jaegertracing.io/docs/1.47/features/#topology-graphs
    • Aggregates data for a given time window (originally at Uber - 15min) and writes a summary snapshot to storage
    • Not easy deployment solution is provided in the repository.
  • Spark Dependencies
    • Batch job that reads all data for a period of time, aggregates, and writes a summary snapshot to storage.
    • Does not require Kafka.
    • Theoretically can be run as frequently as 15min to produce similar results as Flink jobs above, but the implementation for Cassandra may need to be tweaked for that.
    • Does not support transitive dependency graphs.

Objectives:

  • Ideally we want a single code base that supports both types of service dependencies
  • The solution needs to be documented, packaged (e.g. published containers) and easy to deploy (e.g. with docker compose or k8s operator)
  • Supporting both batch (goes directly against span storage) and streaming (reads from Kafka) is nice to have

Contributor guide

Tech stack
gokafkadockerkubernetes
Domain
backenddatacloud
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
5
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Jaeger basicsKafka and streamingDocker/KubernetesDistributed tracing
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
5
Research direction
Investigate the current Jaeger Analytics Flink and Spark Dependencies repositories to understand their architecture. Propose a unified codebase that supports both batch and streaming modes. Evaluate containers and deployment options like Docker Compose and Kubernetes Operator. Discuss with maintainers about the preferred approach.