Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[winml perf] QNN/NPU basic op-tracing produces no profiling CSV

Open
  • Is something wrong?
#1,448 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
cli, machine-learning, python
Domain
cli, machine-learning, performance, tooling

Research direction

The issue is in winml-cli's perf command with --op-tracing basic for QNN/NPU. Start by examining the CLI's tracing implementation, likely in a command/perf.py or similar module. Look for where the CSV is generated and why it might be empty. Check the integration with ONNX Runtime's QNNExecutionProvider and ETW sessions. Run the provided reproduction steps to see the error code 4. 'Done' is when the command produces a profiling CSV for the given models.

Written by the indexing model from the issue text.

Description

bug P1

Summary

QNN basic operator tracing produces no profiling CSV

Priority: P1, confirmed by the release driver on 2026-09-23.
Bugbash reference: 2026-09-22 / BUG-002.
Status: Observed in the first test round; no fix or post-fix regression is recorded. This issue is based on retained test evidence, not a new test run.

Environment

  • Test date: 2026-09-22; findings triaged on 2026-09-23.
  • CLI: winml-cli 0.4.0, installed from a local wheel. The exact source commit/build provenance is not recorded in the test evidence.
  • Hardware: Snapdragon X Elite X1E80100, Qualcomm Hexagon NPU.
  • OS: Windows 11 26H1, build 28000.2956, ARM64.
  • Python: 3.11.15, workspace virtual environment.
  • EP: QNNExecutionProvider from WinML Catalog, package 2.2480.49.0.
  • ONNX Runtime: 1.27.1.202607110137, Windows ML distribution.
  • ONNX: 1.18.0; PyTorch: 2.14.0; Transformers: 4.57.6.
  • Model: catalog microsoft/resnet-50, image-classification, pixel_values FP32 [1, 3, 224, 224].
  • Quantization, where applicable: w8a16, uint8 weights / uint16 activations, 10 calibration samples.

Reproduction

Run in PowerShell with winml-cli 0.4.0 installed. Use a new output directory. Prepare raw w8a16 and compiled inputs with the same artifact chain used in the bugbash:

$results = '.\repro-qnn-optrace'
New-Item -ItemType Directory -Path $results | Out-Null
uv run winml export -m microsoft/resnet-50 -o "$results\resnet.onnx"
uv run winml optimize -m "$results\resnet.onnx" --ep qnn --device npu --disable-ort-graph-optimization -o "$results\resnet-opt-no-ort.onnx"
uv run winml quantize -m "$results\resnet-opt-no-ort.onnx" -p w8a16 --samples 10 --task image-classification --model-id microsoft/resnet-50 -o "$results\resnet-w8a16.onnx"
uv run winml compile -m "$results\resnet-w8a16.onnx" --ep qnn --device npu -o "$results\resnet-compiled.onnx"

Reproduce on the raw quantized model:

uv run winml perf -m "$results\resnet-w8a16.onnx" --ep qnn --device npu --iterations 3 --warmup 1 --op-tracing basic -o "$results\optrace-raw.json"

Reproduce on the compiled model, including with memory monitoring disabled:

uv run winml perf -m "$results\resnet-compiled.onnx" --ep qnn --device npu --iterations 10 --warmup 2 --no-memory --op-tracing basic -o "$results\optrace-compiled.json"

The disabled ORT graph optimization in preparation bypasses a separately recorded standalone optimize failure; it is not a tracing workaround.

Expected result

Produce basic per-operator profiling data for the QNN/NPU benchmark.

Actual result

Both tracing commands exit with code 4 and produce no profiling CSV:

Op-tracing produced no profiling data (no CSV written).

Ordinary perf without --op-tracing basic has passing records for both raw and compiled inputs.

Workaround

No verified workaround that produces operator-tracing data in this bugbash. Ordinary perf runs successfully, but is not equivalent to operator-level profiling.

Scope and evidence

The failure is observed on this host. Root-cause ownership between CLI, WinML/ORT, and QNN EP remains unconfirmed. The CPU-fallback suggestion in the error is not evidence that CPU fallback occurred. ETW session prerequisites/state were not established by this report; do not infer them from the missing CSV.

The following evidence is retained locally by the reporter; these filenames are an index, not uploaded attachments:

  • perf-qnn-optrace.stdout.log
  • perf-qnn-optrace.stderr.log
  • perf-compiled-optrace.stdout.log
  • perf-compiled-optrace.stderr.log
Dominant language
Python
Stars
40
Forks
11
Avg merge
1d 2h
Merged PRs (30d)
59

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/winml-cli

All issues in microsoft/winml-cli

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.