Live API tests intermittently get results without a Result pod

Open
#41 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
testing

Research direction

Start with tests/test_client.py::test_basic and test_results_iterator, then inspect the temp_result fixture and how Result.results selects pods. The issue suggests incomplete responses may report timed-out pods through @timedout or @timedoutpods, but the CI log does not confirm this. Investigate the proposed test-query timeouts and failure diagnostics; done when the intermittent failures are addressed and tests can distinguish a missing answer from a timed-out pod.

Written by the indexing model from the issue text.

Description

tests/test_client.py::test_basic and test_results_iterator failed intermittently in this job (3.13/ubuntu, PR #40):

FAILED tests/test_client.py::test_basic - ValueError: not enough values to unpack (expected 1, got 0)
FAILED tests/test_client.py::test_results_iterator - StopIteration

The same commit passed every job in the pull_request run.

What happened

The temp_result fixture queries '30 deg C in deg F'. Wolfram|Alpha normally sends back 8 pods (sections of the answer), including a primary Result pod (86 °F (degrees Fahrenheit)). In the failing job, the query succeeded (@success: True) but returned only 7 pods, and none of them was Result. As a result, Result.results, which selects pods that are primary or titled Result, was empty.

This is not the location dependence from #39. From US, UK, German and Japanese IP addresses, and with or without location/units, the query always returned the same 8 pods, including Result. Fifteen repeated queries from one place were also all identical.

Likely cause

Wolfram|Alpha drops pods that take too long to compute. A pod whose computation runs past scantimeout (default 3.0s) or podtimeout (default 4.0s) is left out of the response and listed in the @timedout / @timedoutpods attributes. Locally, this query reports @timing of about 3.3s, which is right at that limit. So a slightly slower moment on Wolfram|Alpha's side could plausibly drop the Result pod.

I couldn't confirm this from the CI log, because pytest truncates the fixture repr before the @timedout attribute. It is distinct from #38: there the HTTP request itself times out, while here the request succeeds but the answer is incomplete.

Possible remedies

  • Pass more generous scantimeout / podtimeout in the test queries so all pods are computed. Possibly also add a Client default, in line with the timeout work in #38.
  • When a test gets an incomplete result, include @timedout / @timedoutpods in the assertion message, so failures say why.
  • Optionally, give Result a property that exposes timed-out pods. Then users, and tests, can tell "no answer" apart from "answer timed out".
Dominant language
Python
Stars
154
Forks
26
PR merge metrics
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.