Order-dependent perf CLI test misses Transformers warning in full commands partition
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 64/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- python
- Domain
- testing-qa
Research direction
Start with tests/unit/commands/test_perf_cli.py, especially TestPerfUnifiedPipeline::test_cli_hf_reveals_huggingface_warning_logs_when_verbose and the preceding test named in the reproduction. Run the minimal ordered pytest command first, then inspect the Hugging Face suppression context and logger handling. Done means the warning assertion receives the record in the ordered pair, the full test file, and the commands partition.
Written by the indexing model from the issue text.
Description
Summary
The Windows commands CI partition deterministically fails at
TestPerfUnifiedPipeline::test_cli_hf_reveals_huggingface_warning_logs_when_verbose
when the test runs in normal file/partition order, while the focused test passes.
This reproduces at exact base 79d7baad10f28cd79e65072f78494e8be758f3df and at candidate
4d1998a6befc385d3aff48ae01ca959273727868. The candidate changes only two OWLv2
recipe JSON files, so the failure is not introduced by that delta.
Environment
- Windows 11
- Python 3.11.14
- pytest 9.0.3
- transformers 5.14.1
- isolated checkout and private
TEMP/TMPper run - inherited
PYTHONPATH, pytest, Transformers, HF Hub, progress, WinML provider, and coverage variables cleared before each run
Exact workflow reproduction
uv run pytest tests/unit/commands tests/unit/config tests/unit/build tests/unit/compiler tests/unit/session tests/unit/eval --tb=short --no-cov -m "not e2e and not npu and not gpu"
Result on both exact refs:
tests/unit/commands/test_perf_cli.py::TestPerfUnifiedPipeline::test_cli_hf_reveals_huggingface_warning_logs_when_verbose FAILED [ 25%]
tests\unit\commands\test_perf_cli.py:1149: assert False
The focused target passes in a fresh process:
1 passed in 2.76s
The full tests/unit/commands/test_perf_cli.py file fails at the same assertion.
Minimal ordered reproduction
uv run pytest `
tests/unit/commands/test_perf_cli.py::TestPerfUnifiedPipeline::test_load_model_does_not_forward_export_policy_details `
tests/unit/commands/test_perf_cli.py::TestPerfUnifiedPipeline::test_cli_hf_reveals_huggingface_warning_logs_when_verbose `
--tb=short --no-cov
Result:
test_load_model_does_not_forward_export_policy_details PASSED [ 50%]
test_cli_hf_reveals_huggingface_warning_logs_when_verbose FAILED [100%]
1 failed, 1 passed in 26.81s
Diagnosis
An external pytest diagnostic plugin recorded logging.getLogger("transformers")
without changing the checkout. At session start:
{"effective_level": 30, "handlers": [], "imported": false, "level": 0, "propagate": true}
Immediately after test_load_model_does_not_forward_export_policy_details:
{"effective_level": 30, "handlers": ["StreamHandler"], "imported": true, "level": 30, "propagate": false}
Transformers 5.14.1 _configure_library_root_logger() installs the handler and sets
propagate = false outside CI. WinML's Hugging Face suppression context saves and restores
verbosity, but does not restore the pre-existing logger handler/propagation topology. The later
test emits through logging.getLogger("transformers"); with propagation disabled, pytest's root
caplog handler does not receive the record, so the second assertion fails.
The test or logging utility should isolate/restore the Transformers logger topology, or the
assertion should capture the library logger in a way that is independent of prior import order.
- Dominant language
- Python
- Stars
- 40
- Forks
- 11
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 51
Getting set up
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/winml-cli
-
bug P1
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Maintainers usually reply within 1 day
-
bug P1 triaged
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
microsoft/winml-cli#1097 · 1 comment ·
Maintainers usually reply within 1 day
-
P2 refactor triaged
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day
-
bug P1
Difficulty 4/5 3-5 days Newbie friendliness 45/100
Maintainers usually reply within 1 day
-
bug P1
Difficulty 3/5 1-2 days Newbie friendliness 65/100
Maintainers usually reply within 1 day
All issues in microsoft/winml-cli
Similar issues
-
adr
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
kristofdegrave/homeassistant-smart-charging#1607 ·
Maintainers usually reply within 1 day
-
namespace operations
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
EclipseFdn/open-vsx.org#13665 ·
Maintainers usually reply within 1 day
-
doc good first issue help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
collective/icalendar#1865 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
canonical/opentelemetry-collector-operator#409 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
mozilla/addons-release-tests#1243 ·
Maintainers usually reply within 1 day