Live API tests intermittently get results without a Result pod
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 55/100
Research direction
Start with tests/test_client.py::test_basic and test_results_iterator, then inspect the temp_result fixture and how Result.results selects pods. The issue suggests incomplete responses may report timed-out pods through @timedout or @timedoutpods, but the CI log does not confirm this. Investigate the proposed test-query timeouts and failure diagnostics; done when the intermittent failures are addressed and tests can distinguish a missing answer from a timed-out pod.
Written by the indexing model from the issue text.
Description
tests/test_client.py::test_basic and test_results_iterator failed intermittently in this job (3.13/ubuntu, PR #40):
FAILED tests/test_client.py::test_basic - ValueError: not enough values to unpack (expected 1, got 0)
FAILED tests/test_client.py::test_results_iterator - StopIteration
The same commit passed every job in the pull_request run.
What happened
The temp_result fixture queries '30 deg C in deg F'. Wolfram|Alpha normally sends back 8 pods (sections of the answer), including a primary Result pod (86 °F (degrees Fahrenheit)). In the failing job, the query succeeded (@success: True) but returned only 7 pods, and none of them was Result. As a result, Result.results, which selects pods that are primary or titled Result, was empty.
This is not the location dependence from #39. From US, UK, German and Japanese IP addresses, and with or without location/units, the query always returned the same 8 pods, including Result. Fifteen repeated queries from one place were also all identical.
Likely cause
Wolfram|Alpha drops pods that take too long to compute. A pod whose computation runs past scantimeout (default 3.0s) or podtimeout (default 4.0s) is left out of the response and listed in the @timedout / @timedoutpods attributes. Locally, this query reports @timing of about 3.3s, which is right at that limit. So a slightly slower moment on Wolfram|Alpha's side could plausibly drop the Result pod.
I couldn't confirm this from the CI log, because pytest truncates the fixture repr before the @timedout attribute. It is distinct from #38: there the HTTP request itself times out, while here the request succeeds but the answer is incomplete.
Possible remedies
- Pass more generous
scantimeout/podtimeoutin the test queries so all pods are computed. Possibly also add aClientdefault, in line with the timeout work in #38. - When a test gets an incomplete result, include
@timedout/@timedoutpodsin the assertion message, so failures say why. - Optionally, give
Resulta property that exposes timed-out pods. Then users, and tests, can tell "no answer" apart from "answer timed out".
- Dominant language
- Python
- Stars
- 154
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Similar issues
-
cognition facet:consolidation memory ready size:S
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
-
proposal
Difficulty 1/5 Under an hour Newbie friendliness 68/100
j3ko/agent-plugins#29 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
-
area:cli area:security bug priority:medium
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
Judgment-Pack/judgment-pack-spec#335 ·
Maintainers usually reply within 1 day