prometheus/prometheus

ST: Optimize CPU/memory use for OM1.0 Text CT/ST parsing (`st-storage` and `created-timestamp-zero-ingestion` features)

Open

#14,823 opened on Sep 4, 2024

View on GitHub
 (3 comments) (1 reaction) (0 assignees)Go (64,042 stars) (10,408 forks)batch import
component/scrapinghelp wanted

Description

Proposal

As observed accidently on main and without any feature flag we use a lot of memory for CreatedTimestamp call for OM text. This is .due to deep coping of parser on every CreatedTimestamp, so technically per every series (!). This was a known naive implementation we decided to accept when iterating on this feature in https://github.com/prometheus/prometheus/pull/14356

This issue is to minimize that memory use. We already discussed this, but to put it here:

  • We should see if we can reuse existing parser (not copy it).
  • If we have to copy, copy once and reuse a second parser (e.g. reset it's positions).

IMPORTANT: This is only relevant to OpenMetrics 1.0. We designed OM 2 to not have that problem.

Acceptance Criteria

  • The additional memory use per OM Text scrape with CT feature is minimal (~within +5% max)

Related to https://github.com/prometheus/prometheus/issues/14217

Contributor guide

Tech stack
go
Domain
backendperformance
Issue type
performance
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Go programmingPrometheus codebaseOpenMetrics formatMemory optimization concepts
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60
Research direction
Examine the deep copy of the parser at model/textparse/openmetricsparse.go line 271. Evaluate whether the parser can be reused without copying across multiple CreatedTimestamp calls, or if copying once and resetting positions is feasible. Review linked PR #14356 for context on the original naive implementation.