Database and Table Level Metrics

Feature Request

Is your feature request related to a problem? Please describe:

Many customers use a single TiDB cluster to serve multiple & hybrid payloads (in different databases). Currently TiDB only supports metrics of the whole TiDB cluster or single TiDB instance. Database or table metrics are missing.

Describe the feature you'd like:

For critical metrics like QPS, latency, errors, a more detailed, per-table and per-database metrics is needed.

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

The technical implementation needs to be further investigated. A possible solution can be using Prometheus labels. The corresponding label in memory metrics need to be deleted when table or database is deleted. Notice that there might be a lot of databases and tables and attaching metrics for each one may affect performance, so that it may not be suitable to adapt all metrics. Also histograms and multi-label metrics need to be very carefully considered, since they notably amplify the number of metrics. Another good idea can be allowing users to config what tables and databases are needed.

The new metrics should be added to the Grafana monitor.

Notice that TiDB already have a similar feature: #9151. However it may lead to memory leaks due to always keeping database names in the memory, as well as not work well when database numbers are huge. The new implementation could refine and improve it.