pytest: TestCase failures reported as passing

Tái hiện sự cố bằng ví dụ TestCase được hiển thị và lệnh plugin ciqueue.pytest_report. Đọc hook unittest của pytest được tham chiếu trong issue và hook reporting của ciqueue để kiểm tra thứ tự của chúng cũng như cách xử lý call.excinfo. Được xem là hoàn tất khi một TestCase bị lỗi được báo cáo là failed thay vì passed.

With this test suite:

# test_example.py
from unittest import TestCase

class TestExample(TestCase):
    def test_fail(self):
        assert False

Running works as expected:

$ pytest -p ciqueue.pytest --queue 'redis://localhost:6379/0?worker=0&build=repro&retry=0&timeout=10' test_example.py
========================================================= test session starts =========================================================
platform darwin -- Python 2.7.13, pytest-4.0.1, py-1.7.0, pluggy-0.6.0
...
collected 1 item

test_example.py F

============================================================== FAILURES ===============================================================
________________________________________________________ TestExample.test_fail ________________________________________________________

self = <test_example.TestExample testMethod=test_fail>

    def test_fail(self):
>       assert False
E       AssertionError: assert False

test_example.py:5: AssertionError
====================================================== 1 failed in 0.06 seconds =======================================================

However, reporting shows that the test passed:

$ pytest -p ciqueue.pytest_report --queue 'redis://localhost:6379/0?worker=0&build=repro&retry=0&timeout=10' test_example.py
========================================================= test session starts =========================================================
platform darwin -- Python 2.7.13, pytest-4.0.1, py-1.7.0, pluggy-0.6.0
...
collected 1 item

test_example.py .                                                                                                               [100%]

====================================================== 1 passed in 0.02 seconds =======================================================

This appears to be due to ciqueue's makereport triggering before the one in pytest which is responsible for setting call.excinfo for TestCase-style tests.

From these docs it looks like:

1. the ordering of multiple tryfirst=True hooks is undefined
2. hookwrapper=True can execute code before a tryfirst hook

So, one fix could be to get pytest to switch to using hookwrapper. I figured I'd raise it with you folks first, though, in case it's something that's easier to fix on this end (by making ciqueue execute later, somehow).