[Feature] Implement batch TopN aggregation for historical data on non-hot nodes

Open
#13,286 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
30/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
java

Research direction

The issue names no files, tests, or entry points. Start by reading related issue #13285 and locating the existing TopN aggregation and warm/cold storage workflows; done should include configurable batch tasks, historical time ranges, naming conventions, and post-import aggregation.

Written by the indexing model from the issue text.

Description

database feature
Search before asking
  • I had searched in the issues and found no similar feature requirement.
Description

Related to #13285

Implement a batch processing system that enables TopN aggregation for historical data stored on warm and cold nodes. This feature will support scenarios where historical data is imported and users need to generate TopN aggregations retroactively with flexible configuration options.

Background

Non-hot storage tiers (warm and cold nodes) often contain historical data that may be imported in bulk. Currently, there's no efficient mechanism to perform TopN aggregations on this historical data without impacting real-time operations or requiring manual intervention.

Proposed Features

Develop a batch processing capability that supports:

  • On-demand TopN aggregation through configurable batch tasks
  • Flexible time-range specifications for historical data processing
  • User-defined naming conventions for generated TopN results
  • Support for importing historical data with subsequent batch aggregation processing
Use case

No response

Related issues

No response

Are you willing to submit a pull request to implement this on your own?
  • Yes I am willing to submit a pull request on my own!
Code of Conduct
Dominant language
Java
Stars
25k
Forks
6.6k
Avg merge
12h 1m
Merged PRs (30d)
43

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/skywalking

All issues in apache/skywalking

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.