solana-labs/solana

validator uses more memory than expected when metrics are enabled

Open

#24,319 opened on Apr 13, 2022

View on GitHub
 (26 comments) (0 reactions) (1 assignee)Rust (12,651 stars) (3,950 forks)batch import
do-not-closegood first issue

Description

Problem

80GB difference between RUST_LOG=solana=info and RUST_LOG=solana=warn for RPC nodes.

Proposed Solution

Relevant question.. is the channel that the metrics are sending to really large? Could there be some large amount of outstanding sends which each is taking some memory and state?

Debug and find why. Possibly reduce the occurrence of spammy metrics.

Contributor guide

Tech stack
rust
Domain
performanceobservability
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
Rust programmingbasic understanding of Solana validator architecture
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20
Research direction
Investigate the memory increase when metrics are enabled. Look at the metrics sending channel in the Solana validator code (likely in `validator/src/` or `metrics/`). Profile memory usage with tools like `heaptrack` or `valgrind` to identify where the excess memory is allocated. Check if outstanding sends in the metrics channel are accumulating. Reduce spammy metrics by filtering or batching. Review comments from the issue for additional hints.