Write developer documentation on writing tests
@PGijsbers is already working on this.
Since Jul 16, 2024.
Assessment
This issue has not been assessed yet.
Description
After some reading and testing, I found that I think I want to stick with the following strategy for now:
- Migration testing, which tests the python api responses against the php ones, when set up against the same pre-populated database. This is so we ensure correctness for migration, and have explicit documentation about the differences (in the unit tests, though they should also end up on the migration page)
- Integration testing for expected paths. Simple tests that test the full Python stack, including FastAPI client, dependency injection, and database interactions. For these tests the data will be added to the database on a per-test level. This makes it easy and fast to run tests by any developer- no management of test database state needed. From measurements I did find that creating the TestClient for FastAPI has considerable overhead (approx. 60ms per test). For this reason, we limit this to one test per kind of expected response. E.g., for get_flow: one with a returned flow, one without a result.
- Test database functions directly on a database which have items inserted on a per-test level. This is still surprisingly responsive, so there is no need to mock anything out.
- Test the broader input/output of the different API methods by calling the Python functions directly instead of through the TestClient. This is so we can test more different scenarios, while being fast. It also ensures that if tests fail at the API level (point 1) but not here, we know it is due to changes of FastAPI its configuration. Database calls at this level can be mocked, to better isolate the input/output processing of this function. I have some consideration of not mocking anything out here, to avoid the extra boilerplate. The database fixtures are pretty performant (~2ms).
This is not set in stone. Perhaps we should not even separate tests out this much, and only factoring out the FastAPI layer is enough. If someone with more experience happens to read this, feel free to leave your thoughts :)
- Dominant language
- Python
- Stars
- 16
- Forks
- 50
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from openml/server-api
-
behavior
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
openml/server-api#337 · 1 comment ·
-
proposal
openml/server-api#375 · 2 comments · 1 assignee ·
-
openml/server-api#374 · 2 comments · 1 assignee ·
-
openml/server-api#373 · 1 assignee ·
-
openml/server-api#372 · 1 assignee ·
All issues in openml/server-api
Similar issues
-
essnmx good first issue
Difficulty 1/5 Under an hour Newbie friendliness 95/100
-
[Feature] 奇物选择添加优先级 Open
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
syfoud/Simulated_Scepter#174 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Giskard-AI/giskard-oss#2840 · 1 comment ·
-
A claim comment carrying the issue number is silently declined while the workflow reports success Openarea: repo bug perceived difficulty: 2
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
yeti-platform/yeti#1380 ·