[!NOTE] Maintainer audit (2026-07-15): this is a historical v0.3 reproduction report, not a confirmed defect on current main. Reproduce with a current commit and capture process RSS/heap profiles, goroutine counts, allocator statistics, and repeated-load plateaus before attributing the cause to a specific subsystem.

Describe the bug

Possible memory leak or unbounded memory retention under load in vLLM Semantic Router v0.3

Summary

I am seeing continuous memory growth in vLLM Semantic Router under load. After stopping the load test, CPU usage returns to 0%, but memory usage does not decrease. Re-running the load test causes memory usage to continue increasing.

This still happens after disabling response_api, OpenTelemetry tracing, router_replay, and semantic_cache, so the issue does not appear to be caused by those optional storage/tracing/cache features.

Environment

• vLLM Semantic Router config version: v0.3
• Service version in config: v0.3.0
• CPU: 16 cores
• Memory: 29.9 GB
• Backend protocol: OpenAI-compatible HTTP backend
• Request type: non-streaming chat completion
• Tested endpoint/name: llm_chat_no_stream
• Load test users: 400
• Spawn rate: 30 users/second
• Observed RPS: about 25.7 RPS
• Average latency: about 3946 ms
• Estimated in-flight requests: around 100

What I expected

After stopping the load test, memory usage should either:

1. Decrease close to the idle baseline, or
2. Stay at a stable warm-state high-water mark and not continue increasing across repeated load-test rounds.

What actually happened

Memory keeps increasing and does not go down after the load test is stopped.

Observed memory usage:

After service restart / idle:
  Memory: 5.0 GB / 29.9 GB
  CPU: 0%

During load test, after about 1 minute:
  Memory: 12.5 GB / 29.9 GB
  RPS: 25.7
  Average latency: 3946.64 ms
  Failures: 0

After stopping the load test:
  Memory: 13.2 GB / 29.9 GB
  CPU: 0%

Later, still after load stopped:
  Memory: 14.8 GB / 29.9 GB
  CPU: 0%

CPU returns to 0%, which suggests requests have finished, but memory remains high and continues growing. Repeated load-test runs cause additional memory growth.

Configuration notes

The following optional features were disabled during the test:

global:
  services:
    response_api:
      enabled: false

    observability:
      tracing:
        enabled: false
      metrics:
        enabled: true

    router_replay:
      enabled: false

  stores:
    semantic_cache:
      enabled: false

However, the following core routing/classification features are still enabled:

global:
  router:
    clear_route_cache: true
    streamed_body:
      enabled: false
    skip_processing:
      enabled: false
    model_selection:
      method: knn
      enabled: true

  model_catalog:
    embeddings:
      semantic:
        mmbert_model_path: models/mmbert-embed-32k-2d-matryoshka
        use_cpu: true
        embedding_config:
          model_type: mmbert
          preload_embeddings: true
          target_dimension: 768
          target_layer: 22
          enable_soft_matching: true
          top_k: 1
          min_score_threshold: 0.5

    modules:
      prompt_guard:
        enabled: true
        use_cpu: true
        use_mmbert_32k: true
        model_ref: prompt_guard

      classifier:
        domain:
          threshold: 0.5
          use_cpu: true
          use_mmbert_32k: true
          model_ref: domain_classifier

        pii:
          threshold: 0.9
          use_cpu: true
          use_mmbert_32k: true
          model_ref: pii_classifier

      complexity:
        prototype_scoring:
          enabled: true
          cluster_similarity_threshold: 0.9
          max_prototypes: 8
          best_weight: 0.75
          top_m: 2

      feedback_detector:
        enabled: true
        use_cpu: true
        use_mmbert_32k: true
        model_ref: feedback_detector

Translated routing decision descriptions

The original config had several Chinese descriptions. Here are the English translations used for reporting:

routing:
  decisions:
    - name: jailbreak-intercept-route
      description: When jailbreak content is detected, force route to the blackhole model for blocking.

    - name: low-token-easy-code-route
      description: For code-domain requests that are classified as simple coding questions, route to the lite model.

    - name: low-token-hard-code-route
      description: For code-domain requests that are not classified as simple coding questions, route to the specialized coding model.

    - name: high-token-route
      description: For normal content with a large context, route directly to the pro model.

    - name: safe-default-route
      description: Handle regular short text requests outside the code domain by defaulting to the lite model.

To Reproduce

Steps to reproduce

1. Start vLLM Semantic Router with the attached configuration.

2. Ensure these features are disabled:

   • response_api.enabled: false
   • observability.tracing.enabled: false
   • router_replay.enabled: false
   • semantic_cache.enabled: false

3. Run a non-streaming chat completion load test:

   • 400 users
   • 30 users/second spawn rate
   • About 25 RPS
   • Average latency around 4 seconds

4. Stop the load test.

5. Observe memory usage after CPU returns to 0%.

6. Repeat the load test again without restarting the router process.

Actual result

Memory grows from about 5 GB idle to 12.5 GB during the first load test. After stopping the load test, memory remains around 13.2 GB and later increases to 14.8 GB while CPU is 0%. Repeated load-test runs continue increasing memory usage.

Expected result

Memory should either be released after requests complete, or stabilize at a warm-state high-water mark. It should not continue increasing after load stops or across repeated load-test rounds.

Possible cause

Since tracing, response storage, router replay, and semantic cache are disabled, the issue may be in one of the core routing paths, such as:

• prompt_guard
• domain_classifier
• pii_classifier
• feedback_detector
• complexity.prototype_scoring
• semantic embedding / tokenizer path
• model_selection: knn
• non-streaming request/response buffering because streamed_body.enabled is currently false
• HTTP forwarding or connection pooling

This looks like either a memory leak or unbounded memory retention in the core routing/classification path.

Additional question

Is there a recommended way to limit or release memory used by the CPU-based MMBERT classifiers, semantic embedding model, KNN model selection, or non-streaming request buffering?

Also, should streamed_body.enabled: true be required for high-concurrency non-streaming chat completion proxying?

Expected behavior

See Expected result above: memory should return toward an idle baseline or stabilize at a repeatable warm-state plateau.

Affected layer

Unconfirmed. Reproduction and profiling should isolate router request buffering, classification/model runtime, allocator behavior, and other retained state before assigning ownership.

Additional context

This report describes v0.3. A current-main reproduction with heap/RSS profiles and repeated load/idle cycles is required before treating the listed possible causes as findings.