Performance regression from v0.1.3 to v0.2.0 on RTX4060 8GB

Start by comparing the MoE auto-fetch and bandwidth benchmark selection paths between v0.1.3 and v0.2.0; the report points to ft bench bw profiles under ~/.cache/freetoken/benchbw/*.json and logs where both auto policies find no matching row. Reproduce with the reported RTX 4060 setup and command, then inspect how benchmark results drive the selected MoE policy. Done means v0.2.0 selects a valid policy and the regression is explained or fixed, with the benchmark and serving behavior checked.

Before you start

• I have read the FAQ and my problem is not answered there.
• I have read the Roadmap and this is not already planned there.
• I have searched existing issues and found no duplicate.
• I am on the latest release, or on a freshly rebuilt main when building from source.

What happened

Environment

RTX4060 8GB
i5-8400
RAM 48GB
Windows 11 + WSL2

Model
Qwen3.6-35B-A3B-NVFP4

Working version
v0.1.3

Problem version
v0.2.0

Issue

The same startup parameters and same model are much slower on v0.2.0.

v0.1.3 feels responsive.

v0.2.0 often takes much longer before the first generated token.

I also noticed:

v0.1.3
--moe-hybrid-max-fetch auto: fetching 22.7%

v0.2.0
--moe-hybrid-max-fetch auto: no ft bench bw row
--moe-locked-fetch auto: no ft bench bw row

ft bench bw completes successfully.

Logs below.
Same startup parameters were used on both versions.

Command

ft serve
--model ~/LLM-models/nvidia/Qwen3.6-35B-A3B-NVFP4
--host=0.0.0.0
--port=1919
--memory-ratio 0.87
--moe-strategy hybrid
--moe-cpu-layers auto
--max-running-requests 1
--kv-reserve-tokens 65536
--max-seq-len-override 65536

v0.1.3

freetoken version 0.1.3

[2026-10-24|16:23:42] INFO --moe-cpu-layers: layers [0, 1, 39] settled pageable
[2026-10-24|16:23:42] INFO --moe-cache-auto resolved moe_cache_size=858
[2026-10-24|16:23:42] INFO --moe-hybrid-max-fetch auto: fetching 22.7% of each decode step

[2026-10-24|16:23:45] INFO Free memory after initialization: 0.59 GiB

[2026-10-24|16:23:45] INFO Start capturing CUDA graphs with sizes: [1]

[2026-10-24|16:24:48] INFO Scheduler is idle, waiting for new reqs...
[2026-10-24|16:24:48] INFO API server is ready

Inference example:

[2026-10-24|16:26:39] INFO Prefill batch, #new-token: 8192
[2026-10-24|16:26:39] INFO Prefill batch, #new-token: 8124
[2026-10-24|16:26:41] INFO Decode batch ...

This version feels responsive and usable.

---

v0.2.0

freetoken version 0.2.0

[2026-10-24|17:29:59] WARNING --moe-cpu-layers: layers [0, 13, 27] settled pageable

[2026-10-24|17:30:00] WARNING --moe-hybrid-max-fetch auto: no ft bench bw row ...
[2026-10-24|17:30:00] WARNING --moe-locked-fetch auto: no ft bench bw row ...

[2026-10-24|17:30:00] INFO locked layers [0, 13, 27]: GPU slots + fetch up to ...

[2026-10-24|17:30:02] INFO Free memory after initialization: 0.51 GiB

[2026-10-24|17:30:02] INFO Start capturing CUDA graphs with sizes: [1]

[2026-10-24|17:30:05] INFO Scheduler is idle, waiting for new reqs...
[2026-10-24|17:30:05] INFO API server is ready

Inference example:

[2026-10-24|17:31:56] INFO Prefill batch, #new-token: 8192
[2026-10-24|17:31:56] INFO Prefill batch, #new-token: 6701
[2026-10-24|17:32:05] INFO Decode batch ...

Another example:

[2026-10-24|17:38:56] INFO Prefill batch, #new-token: 8192
[2026-10-24|17:38:57] INFO Prefill batch, #new-token: 6761
[2026-10-24|17:39:16] INFO Decode batch ... gen throughput (token/s): 2.42
[2026-10-24|17:39:20] INFO Decode batch ... gen throughput (token/s): 9.68

I also ran:

ft bench bw

The benchmark completed successfully and produced a profile:

~/.cache/freetoken/benchbw/*.json

However v0.2.0 still reports:

--moe-hybrid-max-fetch auto: no ft bench bw row
--moe-locked-fetch auto: no ft bench bw row

Question:

Is this expected in v0.2.0?
Has the auto MoE policy changed significantly compared to v0.1.3?

How did you install FreeToken

pip / uv wheel

FreeToken version

0.2.0,0.1.3

OS

Windows 11

OS details

26H2, WSL2 Ubuntu-26.04

GPU and driver

RTX4060 LP 8GB NVIDIA-SMI 615.78.05

CPU and system RAM

i5 8400 48GB

Checkpoint

nvidia/Qwen3.6-35B-A3B-NVFP4

Command

ft serve \
  --model ~/LLM-models/nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --host=0.0.0.0 \
  --port=1919 \
  --cors "*" \
  --memory-ratio 0.87 \
  --moe-strategy hybrid \
  --moe-cpu-layers auto \
  --max-running-requests 1 \
  --kv-reserve-tokens 65536 \
  --max-seq-len-override 65536

Full log

~$ bash ft-up.sh
[2026-10-24|16:22:49] INFO     Parsed arguments:
ServerArgs(model_path='/home/testman/LLM-models/nvidia/Qwen3.6-35B-A3B-NVFP4', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, max_running_req=1, attention_backend='auto', moe_strategy='hybrid', quant_backend=None, ple_backend='disk', expert_load='auto', moe_cache_size=0, moe_cache_rate=None, moe_cache_auto=True, kv_reserve_tokens=65536, moe_cache_policy='lru', moe_p
[2026-10-24|16:22:49|FrontendAPI] INFO     Default sampling config (source=model): temperature=1.0, top_k=20, top_p=0.95
INFO:     Started server process [61802]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:1919 (Press CTRL+C to quit)
/home/testman/FreeToken/python/freetoken/engine/engine.py:1135: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead.
  torch.cuda.memory._set_allocator_settings("expandable_segments:True")
[2026-10-24|16:22:53|core|rank=0] INFO     Enabled expandable_segments (override via PYTORCH_ALLOC_CONF)
[2026-10-24|16:22:53|core|rank=0] INFO     Auto-selected attention backend: fi
[2026-10-24|16:22:53|core|rank=0] INFO     Resolved config: moe_strategy='hybrid', attention_backend='fi', cache_type='hybrid_radix', page_size=1
[2026-10-24|16:22:53|core|rank=0] INFO     Free memory before loading model: 6.92 GiB
Loading weights: 100%|███████████████████████████████████████████| 3/3 [00:08<00:00,  2.71s/it]
[2026-10-24|16:23:05|core|rank=0] INFO     MoE experts: nvfp4 via triton
[2026-10-24|16:23:05|core|rank=0] INFO     --moe-cpu-layers auto: banks 16.93 GiB > pin budget 15.67 GiB; locking 3 head+tail MoE layers for CPU decode ([0, 1, 39])
[2026-10-24|16:23:05|core|rank=0] INFO     --moe-cpu-layers split residency: disabling MoE prefill overlap (locked layers prefill via synchronous pageable copies)
[2026-10-24|16:23:05|core|rank=0] INFO     expert banks: slow path (parallel build)
Loading experts (parallel):   0%|                                  | 0.00/21.8G [00:00<?, ?B/s][2026-10-24|16:23:15|core|rank=0] WARNING  bank lock failed; leaving this and later banks pageable: [Errno 12] mlock(0.2 GiB): Cannot allocate memory (RLIMIT_MEMLOCK / `ulimit -l` caps OS-locked bytes; raise it or shrink --moe-cpu-layers)
[2026-10-24|16:23:42|core|rank=0] WARNING  --moe-cpu-layers: layers [0, 1, 39] settled pageable instead of OS-locked (lock failed); they still decode on the CPU executor but may swap under memory pressure
[2026-10-24|16:23:42|core|rank=0] INFO     --moe-cache-auto resolved moe_cache_size=858 num_pages=65556 (prefill_overlap=False)
[2026-10-24|16:23:42|core|rank=0] INFO     --moe-hybrid-max-fetch auto: fetching 22.7% of each decode step's expert misses over PCIe (benched PCIe/CPU bandwidth ratio), the rest on the CPU
[2026-10-24|16:23:42|core|rank=0] INFO     torch intra-op threads: 6 -> 1 (cores reserved for the pinned CPU MoE pool)
[2026-10-24|16:23:42|core|rank=0] INFO     CPU MoE executor ready: threads=5 (pinned to cores 0..4) isa=avx2 fmt=nvfp4 H=2048 I=512 experts=256 layers=40 top_k=8 act=silu max_tokens=1
[2026-10-24|16:23:42|core|rank=0] INFO     Multimodal enabled: QwenVLMMProcessor, encoders ['vision'] on host, serving ['image']
[2026-10-24|16:23:45|core|rank=0] INFO     Allocating 65556 tokens for KV cache, K + V = 1.25 GiB
[2026-10-24|16:23:45|core|rank=0] INFO     Free memory after initialization: 0.59 GiB
[2026-10-24|16:23:45|core|rank=0] INFO     Start capturing CUDA graphs with sizes: [1]
[2026-10-24|16:23:45|core|rank=0] INFO     Free GPU memory before capturing CUDA graphs: 0.59 GiB
Capturing graphs: bs = 1   | avail_mem = 0.60 GiB: 100%|██████| 1/1 [01:02<00:00, 62.82s/batch]
[2026-10-24|16:24:48|core|rank=0] INFO     Free GPU memory after capturing CUDA graphs: 0.60 GiB
INFO:     127.0.0.1:44652 - "POST /v1/chat/completions HTTP/1.1" 503 Service Unavailable
[2026-10-24|16:24:48|core|rank=0] INFO     Scheduler is idle, waiting for new reqs...
[2026-10-24|16:24:48|FrontendAPI] INFO     API server is ready to serve on 0.0.0.0:1919
INFO:     127.0.0.1:44652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:48722 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-24|16:25:12|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.25, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 0, #queue-req: 1, input throughput (token/s): 344.66
[2026-10-24|16:25:13|FrontendAPI] WARNING  Aborting request for user 0
INFO:     127.0.0.1:48736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-24|16:25:46|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.25, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 0, #queue-req: 1, input throughput (token/s): 390.50
[2026-10-24|16:25:57|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.37, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 0, #queue-req: 1, input throughput (token/s): 757.39
[2026-10-24|16:26:03|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.41, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 1, #queue-req: 0, input throughput (token/s): 1445.60
[2026-10-24|16:26:03|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 2309, #cached-token: 0, token usage: 0.41, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 0, input throughput (token/s): 38244.00
[2026-10-24|16:26:05|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 26926, token usage: 0.41, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 0.52, #queue-req: 0
[2026-10-24|16:26:07|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 26966, token usage: 0.41, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 19.02, #queue-req: 0
[2026-10-24|16:26:09|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 27006, token usage: 0.41, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 20.04, #queue-req: 0
[2026-10-24|16:26:11|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 27046, token usage: 0.41, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 18.26, #queue-req: 0
[2026-10-24|16:26:14|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 27086, token usage: 0.41, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 18.43, #queue-req: 0
[2026-10-24|16:26:16|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 27126, token usage: 0.41, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 18.06, #queue-req: 0
[2026-10-24|16:26:18|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 27166, token usage: 0.41, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 19.70, #queue-req: 0
INFO:     127.0.0.1:56978 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-24|16:26:18|core|rank=0] INFO     Scheduler is idle, waiting for new reqs...
INFO:     127.0.0.1:56982 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:56988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-24|16:26:39|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.25, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 1, #queue-req: 0, input throughput (token/s): 225.77
[2026-10-24|16:26:39|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8124, #cached-token: 0, token usage: 0.25, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 0, input throughput (token/s): 49755.67
[2026-10-24|16:26:41|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 16349, token usage: 0.25, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 1.71, #queue-req: 0
[2026-10-24|16:26:43|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 16389, token usage: 0.25, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 19.60, #queue-req: 0
[2026-10-24|16:26:44|core|rank=0] INFO     Scheduler is idle, waiting for new reqs...
[2026-10-24|16:26:48|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 452, #cached-token: 0, token usage: 0.01, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 0, input throughput (token/s): 50.90
INFO:     127.0.0.1:40644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:52608 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-24|16:26:49|core|rank=0] INFO     Scheduler is idle, waiting for new reqs...
INFO:     127.0.0.1:34146 - "GET /api/v1/models HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:34146 - "GET /api/tags HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:34146 - "GET /v1/props HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:34146 - "GET /props HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:34146 - "GET /version HTTP/1.1" 404 Not Found
[2026-10-24|16:24:23] INFO     reasoning-effort profile: supported=['high', 'low', 'max', 'medium', 'minimal', 'none', 'xhigh'] default=None
INFO:     127.0.0.1:34154 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:34160 - "POST /v1/chat/completions HTTP/1.1" 503 Service Unavailable
INFO:     127.0.0.1:34160 - "POST /v1/chat/completions HTTP/1.1" 503 Service Unavailable
INFO:     127.0.0.1:34160 - "POST /v1/chat/completions HTTP/1.1" 503 Service Unavailable
INFO:     127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 503 Service Unavailable
[2026-10-24|16:25:25|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.37, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 0, #queue-req: 1, input throughput (token/s): 625.56
[2026-10-24|16:25:25|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.00, #mamba-slot: 0/8, mamba usage: 0.00, #running-req: 0, #queue-req: 1, input throughput (token/s): 38360.12
^C
[2026-10-24|16:43:24|core|rank=0] INFO     Scheduler exiting gracefully...
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [61802]
/usr/lib/python3.14/multiprocessing/resource_tracker.py:396: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown: {'/mp-1slhvi86'}
  warnings.warn(
~$ 
~$ 
~$ bash ft200-up.sh
[2026-10-24|16:55:24] INFO     Parsed arguments:
ServerArgs(model_path='/home/testman/LLM-models/nvidia/Qwen3.6-35B-A3B-NVFP4', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, hf_overrides={}, max_running_req=1, attention_backend='auto', moe_strategy='hybrid', quant_backend=None, ple_backend='disk', swa_decoder_replay='bounded', expert_load='auto', moe_cache_size=0, moe_cache_rate=None, moe_cache_auto=True, moe_cache_mem=None, moe_cache_scope='unified', moe_cache_split='equal', kv_reserve_tokens=65536, moe_cache_policy='lru', moe_prefill_overlap=True, moe_prefill_hit_d2d=False, moe_collect_stats=False, moe_cpu_threads=0, moe_cpu_layers='auto', moe_hybrid_max_fetch=-1, moe_locked_fetch=-1.0, moe_locked_max_fetch=8, cuda_graph_bs=None, cuda_graph_max_bs=None, page_size=1, memory_ratio=0.87, linear_state_cache_ratio=2.0, swa_full_tokens_ratio=0.2, swa_num_pages_override=None, distributed_timeout=60.0, use_dummy_weight=False, use_pynccl=True, max_seq_len_override=65536, num_page_override=None, num_token_override=None, mm=MultimodalConfig(disabled_encoders=frozenset(), embed_cache_device='cpu', encoder_weights='host', image_min_tokens=None, image_max_tokens=None, processor_kwargs={}), max_extend_tokens=8192, cache_type='radix', offline_mode=False, decode_log_interval=40, special_token_ckpt=False, _unique_suffix='.pid=63022', server_host='0.0.0.0', server_port=1919, num_tokenizer=0, silent_output=False, shell_mode=False, served_model_name='Qwen3.6-35B-A3B-NVFP4', tool_call_parser='qwen3_coder', reasoning_parser='qwen3', sampling_defaults='model', max_output_tokens=None, enable_cache_report=False, anthropic_inline_system='auto', allowed_media_domains='', allowed_local_media_path='', cors_origins='*', gpu=(), gpu_assigned=None)
[2026-10-24|16:55:24|FrontendAPI] INFO     Default sampling config (source=model): temperature=1.0, top_k=20, top_p=0.95
INFO:     Started server process [63022]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:1919 (Press CTRL+C to quit)
/home/testman/ft200/python/freetoken/engine/engine.py:1370: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead.
  torch.cuda.memory._set_allocator_settings("expandable_segments:True")
[2026-10-24|16:55:28|core|rank=0] INFO     Enabled expandable_segments (override via PYTORCH_ALLOC_CONF)
[2026-10-24|16:55:28|core|rank=0] INFO     Auto-selected attention backend: fi
[2026-10-24|16:55:28|core|rank=0] INFO     Resolved config: moe_strategy='hybrid', attention_backend='fi', cache_type='hybrid_radix', page_size=1
[2026-10-24|16:55:28|core|rank=0] INFO     Free memory before loading model: 6.92 GiB
[2026-10-24|16:55:28|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:28|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:28|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
Loading weights: 100%|███████████████████████████████████████████| 3/3 [00:09<00:00,  3.05s/it]
[2026-10-24|16:55:42|core|rank=0] INFO     MoE experts: nvfp4 via triton
[2026-10-24|16:55:43|core|rank=0] INFO     expert banks: slow path (parallel build)
[2026-10-24|16:56:25|core|rank=0] INFO     --moe-cpu-layers auto (pin probe): pinned 40 MoE layers (16.99 GiB) in 1.1s, locked 0 []
[2026-10-24|16:56:25|core|rank=0] INFO     --moe-cache-auto resolved moe_cache_size=854 num_pages=65544 (prefill_overlap=True)
[2026-10-24|16:56:26|core|rank=0] INFO     torch intra-op threads: 6 -> 1 (cores reserved for the pinned CPU MoE pool)
[2026-10-24|16:56:26|core|rank=0] INFO     CPU MoE executor ready: threads=5 (pinned to cores 0..4) isa=avx2 fmt=nvfp4: 40 layers H=2048 I=512 experts=256 layers=40 top_k=8 act=silu max_tokens=1 spin_wait=False handshake=kernel
[2026-10-24|16:56:26|core|rank=0] WARNING  --moe-hybrid-max-fetch auto: no `ft bench bw` row for 40 MoE layers (nvfp4 2048x512) at 5 CPU threads; using a fixed fetch cap of 1 there. `ft bench bw --model <checkpoint>` benches each layer kind
[2026-10-24|16:56:26|core|rank=0] INFO     Multimodal enabled: QwenVLMMProcessor, encoders ['vision'] on host, serving ['image']
In file included from /home/testman/.local/share/uv/python/cpython-3.11.16-linux-x86_64-gnu/include/python3.11/Python.h:12,
                 from /home/testman/ft200/.venv/lib/python3.11/site-packages/triton/backends/nvidia/driver.c:9:
/home/testman/.local/share/uv/python/cpython-3.11.16-linux-x86_64-gnu/include/python3.11/pyconfig.h:1796:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1796 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /home/testman/ft200/.venv/lib/python3.11/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /home/testman/ft200/.venv/lib/python3.11/site-packages/triton/backends/nvidia/driver.c:1:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
[2026-10-24|16:56:30|core|rank=0] INFO     Allocating 65544 tokens for KV cache, K + V = 1.25 GiB
[2026-10-24|16:56:30|core|rank=0] INFO     Free memory after initialization: 0.51 GiB
[2026-10-24|16:56:30|core|rank=0] INFO     Start capturing CUDA graphs with sizes: [1]
[2026-10-24|16:56:30|core|rank=0] INFO     Free GPU memory before capturing CUDA graphs: 0.64 GiB
Capturing graphs: bs = 1   | avail_mem = 0.62 GiB: 100%|██████| 1/1 [01:17<00:00, 77.83s/batch]
[2026-10-24|16:57:48|core|rank=0] INFO     Free GPU memory after capturing CUDA graphs: 0.64 GiB
[2026-10-24|16:57:49|core|rank=0] INFO     Scheduler is idle, waiting for new reqs...
[2026-10-24|16:57:49|FrontendAPI] INFO     API server is ready to serve on 0.0.0.0:1919
INFO:     127.0.0.1:40270 - "GET /api/v1/models HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:40270 - "GET /api/tags HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:40270 - "GET /v1/props HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:40270 - "GET /props HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:40280 - "GET /api/v1/models HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:40280 - "GET /api/tags HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:40270 - "GET /version HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:40280 - "GET /v1/props HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:40280 - "GET /props HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:40280 - "GET /version HTTP/1.1" 404 Not Found
[2026-10-24|16:58:26] INFO     reasoning-effort profile: supported=['high', 'low', 'max', 'medium', 'minimal', 'none', 'xhigh'] default=None
INFO:     127.0.0.1:40290 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:40302 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:40326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:40310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
/home/testman/ft200/.venv/lib/python3.11/site-packages/triton/language/core.py:2625: UserWarning: tl.make_block_ptr is deprecated. Use TensorDescriptor or tl.make_tensor_descriptor instead.
  warn("tl.make_block_ptr is deprecated. Use TensorDescriptor or tl.make_tensor_descriptor instead.")
[2026-10-24|17:02:19|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.23, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 1, #queue-req: 1, input throughput (token/s): 30.28
[2026-10-24|17:02:22|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 6663, #cached-token: 0, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 1, input throughput (token/s): 2124.69
[2026-10-24|17:02:26|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 14896, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 0.14, #queue-req: 1
INFO:     127.0.0.1:48590 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:48600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-24|17:04:42|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.23, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 1, #queue-req: 1, input throughput (token/s): 58.82
[2026-10-24|17:04:43|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 6917, #cached-token: 0, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 1, input throughput (token/s): 5123.38
[2026-10-24|17:04:46|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 15146, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 0.28, #queue-req: 1
[2026-10-24|17:04:50|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 15186, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 12.14, #queue-req: 1
/home/testman/ft200/.venv/lib/python3.11/site-packages/triton/language/core.py:2625: UserWarning: tl.make_block_ptr is deprecated. Use TensorDescriptor or tl.make_tensor_descriptor instead.
  warn("tl.make_block_ptr is deprecated. Use TensorDescriptor or tl.make_tensor_descriptor instead.")
INFO:     127.0.0.1:38910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-24|17:04:59|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 57, #cached-token: 14899, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 1, input throughput (token/s): 3.63
[2026-10-24|17:05:00|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 14970, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 3.86, #queue-req: 1
INFO:     127.0.0.1:57506 - "GET /api/v1/models HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:57506 - "GET /api/tags HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:57506 - "GET /v1/props HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:57506 - "GET /props HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:57506 - "GET /version HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:57510 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:57520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-24|17:05:15|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 1188, #cached-token: 15104, token usage: 0.25, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 2, input throughput (token/s): 73.64
[2026-10-24|17:05:18|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 16327, token usage: 0.25, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 2.19, #queue-req: 2
[2026-10-24|17:05:22|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 16367, token usage: 0.25, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 11.21, #queue-req: 2
[2026-10-24|17:05:25|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 16407, token usage: 0.25, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 11.41, #queue-req: 2
[2026-10-24|17:05:29|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 16447, token usage: 0.25, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 11.04, #queue-req: 2
[2026-10-24|17:05:33|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 16487, token usage: 0.25, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 10.95, #queue-req: 2
[2026-10-24|17:05:36|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 16527, token usage: 0.25, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 10.55, #queue-req: 2
[2026-10-24|17:05:40|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 16567, token usage: 0.25, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 11.61, #queue-req: 2
[2026-10-24|17:05:43|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 16607, token usage: 0.25, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 11.90, #queue-req: 2
[2026-10-24|17:05:47|core|rank=0] INFO     Decode batch, #running-req: 0, #token: 0, token usage: 0.00, #mamba-slot: 0/8, mamba usage: 0.00, gen throughput (token/s): 11.34, #queue-req: 2
INFO:     127.0.0.1:44090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-24|17:05:53|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 259, #cached-token: 0, token usage: 0.00, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 2, input throughput (token/s): 6.87
INFO:     127.0.0.1:56380 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:56384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-24|17:08:10|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.23, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 1, #queue-req: 2, input throughput (token/s): 59.80
[2026-10-24|17:08:11|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 6991, #cached-token: 0, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 2, input throughput (token/s): 5149.64
[2026-10-24|17:08:13|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 15210, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 0.27, #queue-req: 2
INFO:     127.0.0.1:33914 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:33938 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:33946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-24|17:08:22|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 333, #cached-token: 16647, token usage: 0.26, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 3, input throughput (token/s): 29.08
[2026-10-24|17:08:23|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 16982, token usage: 0.26, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 4.33, #queue-req: 3
[2026-10-24|17:10:36|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.23, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 1, #queue-req: 3, input throughput (token/s): 61.26
[2026-10-24|17:10:37|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 6701, #cached-token: 0, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 3, input throughput (token/s): 5110.94
INFO:     127.0.0.1:53252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-24|17:10:48|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 452, #cached-token: 0, token usage: 0.01, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 3, input throughput (token/s): 43.97
[2026-10-24|17:10:48|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 457, token usage: 0.01, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 0.27, #queue-req: 3
[2026-10-24|17:13:03|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.23, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 1, #queue-req: 2, input throughput (token/s): 60.39
[2026-10-24|17:13:05|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 6688, #cached-token: 0, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 2, input throughput (token/s): 5033.64
[2026-10-24|17:13:13|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 496, #cached-token: 0, token usage: 0.01, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 2, input throughput (token/s): 61.26
[2026-10-24|17:15:26|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.23, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 1, #queue-req: 1, input throughput (token/s): 61.40
[2026-10-24|17:15:28|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 6781, #cached-token: 0, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 1, input throughput (token/s): 5161.81
[2026-10-24|17:15:29|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 14988, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 0.14, #queue-req: 1
[2026-10-24|17:15:32|core|rank=0] INFO     Decode batch, #running-req: 0, #token: 0, token usage: 0.00, #mamba-slot: 0/8, mamba usage: 0.00, gen throughput (token/s): 12.27, #queue-req: 1
INFO:     127.0.0.1:38064 - "GET /api/v1/models HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:38064 - "GET /api/tags HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:38064 - "GET /v1/props HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:38064 - "GET /props HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:38064 - "GET /version HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:38066 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:38080 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:38086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-24|17:15:39|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 455, #cached-token: 0, token usage: 0.01, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 2, input throughput (token/s): 39.52
[2026-10-24|17:15:46|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 263, #cached-token: 0, token usage: 0.00, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 1, input throughput (token/s): 37.43
INFO:     127.0.0.1:38070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:44540 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-24|17:18:01|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.23, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 1, #queue-req: 0, input throughput (token/s): 60.64
[2026-10-24|17:18:02|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 6894, #cached-token: 0, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 0, input throughput (token/s): 5210.51
[2026-10-24|17:18:04|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 15102, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 0.26, #queue-req: 0
[2026-10-24|17:18:08|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 15142, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 10.93, #queue-req: 0
[2026-10-24|17:18:10|core|rank=0] INFO     Scheduler is idle, waiting for new reqs...
[2026-10-24|17:18:17|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 452, #cached-token: 0, token usage: 0.01, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 0, input throughput (token/s): 30.61
INFO:     127.0.0.1:60972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-24|17:18:18|core|rank=0] INFO     Scheduler is idle, waiting for new reqs...
^C
[2026-10-24|17:23:12|core|rank=0] INFO     Scheduler exiting gracefully...
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [63022]
~$ /home/testman/.local/share/uv/python/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

~$

Anything else

No response