Add Prometheus Metrics Endpoint

🎯 Goal

Implement Prometheus metrics endpoint to enable monitoring and observability of the API, tracking key performance indicators like request counts, latency, error rates, and resource usage.

📋 Description

Add a /metrics endpoint that exposes application metrics in Prometheus format. This enables integration with monitoring tools like Prometheus and Grafana for production observability and alerting.

✅ Acceptance Criteria

1. Install Prometheus Client

• Add dependency to go.mod:

go get github.com/prometheus/client_golang/prometheus
go get github.com/prometheus/client_golang/prometheus/promhttp

2. Create Metrics Middleware

• Create new file: internal/middleware/metrics.go
• Track these metrics:
  • HTTP requests total (counter) - by method, path, status
  • HTTP request duration (histogram) - in seconds
  • HTTP requests in progress (gauge) - concurrent requests
  • HTTP request size (histogram) - in bytes
  • HTTP response size (histogram) - in bytes

3. Metrics Implementation

Define Metrics

var (
    httpRequestsTotal = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_total",
            Help: "Total number of HTTP requests",
        },
        []string{"method", "endpoint", "status"},
    )
    
    httpRequestDuration = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "http_request_duration_seconds",
            Help:    "HTTP request latency in seconds",
            Buckets: prometheus.DefBuckets,
        },
        []string{"method", "endpoint"},
    )
    
    httpRequestsInProgress = prometheus.NewGauge(
        prometheus.GaugeOpts{
            Name: "http_requests_in_progress",
            Help: "Current number of HTTP requests being processed",
        },
    )
)

Register Metrics

• Register all metrics in init() function
• Handle registration errors properly

4. Metrics Middleware

• Create middleware that:
  • Increments in-progress gauge before request
  • Decrements in-progress gauge after request
  • Records request duration
  • Increments request counter with labels
  • Tracks request/response sizes

Example structure:

func PrometheusMiddleware() gin.HandlerFunc {
    return func(c *gin.Context) {
        start := time.Now()
        httpRequestsInProgress.Inc()
        
        c.Next()
        
        duration := time.Since(start).Seconds()
        status := strconv.Itoa(c.Writer.Status())
        
        httpRequestsTotal.WithLabelValues(c.Request.Method, c.FullPath(), status).Inc()
        httpRequestDuration.WithLabelValues(c.Request.Method, c.FullPath()).Observe(duration)
        httpRequestsInProgress.Dec()
    }
}

5. Metrics Endpoint

• Add /metrics endpoint in internal/server/router.go
• Use promhttp.Handler() to serve metrics
• Endpoint should be accessible without authentication
• Exclude /metrics from metrics collection (avoid infinite loop)

router.GET("/metrics", gin.WrapH(promhttp.Handler()))

6. Additional Application Metrics (Optional but Recommended)

• Database connection pool metrics
• Active users (gauge)
• Failed login attempts (counter)
• JWT token generation/validation (counter)

7. Docker Integration

• Update docker-compose.yml to include Prometheus service:

prometheus:
  image: prom/prometheus:latest
  ports:
    - "9090:9090"
  volumes:
    - ./prometheus.yml:/etc/prometheus/prometheus.yml
  command:
    - '--config.file=/etc/prometheus/prometheus.yml'

• Create prometheus.yml configuration:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'grab-api'
    static_configs:
      - targets: ['app:8080']

8. Grafana Dashboard (Optional)

• Add Grafana service to docker-compose.yml
• Create basic dashboard JSON
• Document dashboard import process

9. Configuration

• Add metrics configuration to configs/config.yaml:
  • METRICS_ENABLED (bool) - Enable/disable metrics
  • METRICS_PATH (string) - Metrics endpoint path (default: "/metrics")

10. Testing

• Verify /metrics endpoint returns Prometheus format
• Test metrics are updated after requests
• Verify counter increases correctly
• Test histogram buckets
• Add unit tests for metrics middleware

11. Documentation

• Update README.md with metrics information
• Document how to access Prometheus UI (http://localhost:9090)
• Add example Prometheus queries
• Document Grafana setup (if included)
• Add architecture diagram showing observability stack

💡 Implementation Hints

Middleware Registration

In internal/server/router.go:

// Register metrics middleware
if config.MetricsEnabled {
    router.Use(middleware.PrometheusMiddleware())
}

// Metrics endpoint (should be before auth middleware)
router.GET("/metrics", gin.WrapH(promhttp.Handler()))

Exclude Health and Metrics from Metrics

if c.FullPath() == "/health" || c.FullPath() == "/metrics" {
    c.Next()
    return
}
// ... record metrics

Example Prometheus Queries

# Request rate
rate(http_requests_total[5m])

# Error rate (5xx errors)
rate(http_requests_total{status=~"5.."}[5m])

# 95th percentile latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Requests in progress
http_requests_in_progress

Testing Metrics Endpoint

# Start the API
make up

# Make some requests
curl http://localhost:8080/api/v1/auth/register -X POST -d '{"email":"test@test.com","password":"pass123","name":"Test"}'

# Check metrics
curl http://localhost:8080/metrics

# Should see output like:
# http_requests_total{method="POST",endpoint="/api/v1/auth/register",status="201"} 1
# http_request_duration_seconds_bucket{method="POST",endpoint="/api/v1/auth/register",le="0.005"} 1

📚 Resources

• Prometheus Go Client Documentation
• Prometheus Best Practices
• Gin Prometheus Middleware Example
• Grafana Dashboard Examples
• Monitoring REST APIs

🎓 Difficulty Level

Intermediate - Requires understanding of metrics, middleware, and monitoring concepts. Great for learning observability!

---

Note: This feature is crucial for production monitoring. Test thoroughly with make test and verify metrics are collected correctly. Consider performance impact of metrics collection!