Add troubleshooting guide for timeseries with NaN

Open
#164 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
45/100
Issue type
Documentation
Clarity
Mostly clear
Activity status
Stale
Tech stack
prometheus

Research direction

Start with the Prometheus expressions and examples in the issue, then identify the repository documentation location for troubleshooting guidance. Document how instrumentation can avoid NaN values and how dashboard or alert exprs should handle potential NaNs; done means both cases are covered with clear examples.

Written by the indexing model from the issue text.

Description

Via @jameinel via matrix.

When timeseries have NaNs,

avg_over_time(
  juju_apiserver_request_duration_seconds{
    juju_controller="$controller",
    juju_unit=~"$controller_host",
    method="FullStatus",
    quantile="0.99",error_code!~"not found|unauthorized access"
  }[5m]
)
Image

then aggregation operations produce unexpected results:

avg without (version, error_code) 
(
  juju_apiserver_request_duration_seconds{
    juju_controller="$controller",
    juju_unit=~"$controller_host",
    method="FullStatus",
    quantile="0.99",error_code!~"not found|unauthorized access"
  }[5m]
)
Image

and only when we filter out the NaNs with e.g.

avg(avg_over_time((juju_apiserver_request_duration_seconds > 0.0001)[1h:]))

then we get the expected results.

We should document:

  • How to avoid NaNs at instrumentation time.
  • Treating potential NaN in dashboard/alert exprs.
Dominant language
HCL
Stars
9
Forks
12
Avg merge
1d 10h
Merged PRs (30d)
11

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from canonical/observability-stack

All issues in canonical/observability-stack

Similar issues

More Documentation issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.