sgl-project/sglang

[Bug] The Prometheus metric `avg_request_queue_latency` is not being collected

Open

#6,357 opened on 2025年5月16日

GitHub で見る
 (4 comments) (1 reaction) (0 assignees)Python (6,216 forks)auto 404
good first issue

Repository metrics

Stars
 (28,442 stars)
PR merge metrics
 (平均マージ 2d 1h) (30d で 1,000 merged PRs)

説明

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
  • 5. Please use English, otherwise it will be closed.

Describe the bug

I have integrate Sglang metrics with Grafana dashboards, and the metric avg_request_queue_latency is not currently available, while all the other metrics are being collected.

The avg_request_queue_latency is of importance for understanding the behavior of the system in high load situations.

Reproduction

You can query Prometheus metrics at http://localhost:3000/metrics

Environment

I have deployed using docker compose:

sglang:
    container_name: sglang
    profiles: [chat_completions_sglang]
    image: lmsysorg/sglang:latest
    volumes:
      - ${HOME}/.cache/huggingface:/root/.cache/huggingface
    restart: always
    network_mode: host # required by RDMA
    privileged: true # required by RDMA
    environment:
      HF_TOKEN: ${HF_TOKEN}
    entrypoint: python3 -m sglang.launch_server
    command:
      --model-path ${SGLANG_MODEL_PATH} --tp 8 --enable-dp-attention --dp 8 --trust-remote-code --max-running-requests 128 --enable-metrics
      --host 0.0.0.0
      --port 3000
    ulimits:
      memlock: -1
      stack: 67108864
    ipc: host
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:3000/health || exit 1"]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["0"]
              capabilities: [gpu]

コントリビューターガイド