Graylog2/graylog2-server

Discrepancy in display counts on Sources view

Open

#6502 opened on Sep 27, 2019

View on GitHub
 (2 comments) (0 reactions) (0 assignees)Java (6,945 stars) (1,032 forks)batch import
featuregood first issuetriaged

Description

I have noticed when ingesting backlog(older timestamped data) that the "Messages per minute" line graph and "sources" data do not line up.

The Messages per minute appear to be correct for the ingest rate, but the sources breakdown below it only show messages for each type from within the time window via timestamp. This means in the last hour if you've ingested logs from 2 days ago, the data is not represented as "sources within the last hour".

I would prefer the log sources overview to use relative time to current, not relative time to the log sources for determining sources, if that makes sense.

Expected Behavior

Ingesting logs with a timestamp of 2 days ago within the recent hour should be represented in the sources count for the relative "1 hour" window.

Current Behavior

The sources display only shows sources which have a message timestamp within the last hour. This means anything older than the relative time window is not displayed- even though it was actually ingested during the past hour.

Possible Solution

Use indexed time or another method to query sources in the last hour rather than message timestamp field.

Steps to Reproduce (for bugs)

  1. Rapidly ingest older log data
  2. Look at sources page.

Your Environment

  • Graylog Version: 3.1.2
  • Elasticsearch Version: 6.8

Contributor guide

Tech stack
javaelasticsearch
Domain
backendobservabilitysearchanalytics
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
Basic understanding of Graylog's data modelFamiliarity with Elasticsearch query DSLExperience with Java backend development
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60
Research direction
Examine the code that generates the sources breakdown and the time window query. Look for the differences between the 'Messages per minute' query and the sources query. The issue suggests using indexed time (ingestion time) instead of message timestamp. Investigate the relevant source files in the graylog2 server repository, particularly the parts handling the sources view data aggregation. Consider how the current query filters by message timestamp and how it could be changed to filter by indexed time.