MetricFamilyToOpenMetrics produces output that cannot be parsed

Open
#319 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
go
Domain
api, backend

Research direction

Start in the expfmt package with MetricFamilyToOpenMetrics, FinalizeOpenMetrics, and parser.TextToMetricFamilies, using the reproducer in the issue as the first test to run. Trace how the _total suffix is handled during serialization and parsing. Done means the reported OpenMetrics round trip no longer produces the conflicting HELP error, with the handling of duplicate short names clarified.

Written by the indexing model from the issue text.

Description

I am debugging https://github.com/prometheus/jmx_exporter/issues/621, and I found I can reproduce the root cause of the error with the following test in github.com/prometheus/common/expfmt:

package expfmt

import (
	"bytes"
	"strings"
	"testing"
)

func TestJvmClassesLoaded(t *testing.T) {
	in := `
# HELP jvm_classes_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_loaded gauge
jvm_classes_loaded 6129.0
# HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
# TYPE jvm_classes_loaded_total counter
jvm_classes_loaded_total 6143.0
`
	metricFamilies, err := parser.TextToMetricFamilies(strings.NewReader(in))
	if err != nil {
		t.Error(err)
	}
	openMetricsOutput := bytes.NewBuffer(make([]byte, 0))
	for _, metricFamily := range metricFamilies {
		if _, err := MetricFamilyToOpenMetrics(openMetricsOutput, metricFamily); err != nil {
			t.Fatal(err)
		}
	}
	if _, err := FinalizeOpenMetrics(openMetricsOutput); err != nil {
		t.Fatal(err)
	}
	_, err = parser.TextToMetricFamilies(strings.NewReader(openMetricsOutput.String()))
	if err != nil {
		t.Error(err)
	}
}

It fails with

=== RUN   TestJvmClassesLoaded
    jvm_classes_loaded_test.go:33: text format parsing error in line 4: second HELP line for metric name "jvm_classes_loaded"
--- FAIL: TestJvmClassesLoaded (0.00s)
FAIL
FAIL    github.com/prometheus/common/expfmt     0.002s
FAIL

As the error message indicates, the reason is that the _total suffix is truncated in HELP and TYPE (as defined in #214), and the resulting "short name" conflicts with the name of the gauge.

Two full serialization steps are needed to trigger the error (OpenMetrics format -> Go objects -> OpenMetrics format -> Go objects), so this might seem like an issue that doesn't occur in practice. However, in https://github.com/prometheus/jmx_exporter/issues/621 this happens because of the following setup:

jmx_exporter -> exporter_exporter -> Prometheus server

I'm not sure what to do about this. Should exporters prevent metrics from having the same "short name"?

Dominant language
Go
Stars
297
Forks
380
Avg merge
3d 18h
Merged PRs (30d)
12

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from prometheus/common

All issues in prometheus/common

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.