Add integration tests for HyP3 API job validation

Open
  • Is something wrong?
#2,740 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
testing

Research direction

Start by reading the job specifications and existing validator tests; the issue does not name specific job spec or test files. Add integration coverage for each job type that checks API responses, validator execution order, and the CMR querying and filtering path. Done means comprehensive tests cover these behaviors across the job types.

Written by the indexing model from the issue text.

Description

Jira Task

Jira: https://asfdaac.atlassian.net/browse/TOOL-3663

Note: The above link is accessible only to members of ASF.

We've historically done most/all of our integration testing for HyP3 API job validators in test/sandbox deployments and then summarized the results in PR/issue descriptions. We've treated validator functions as modular and independent of job type and typically only add comprehensive tests for the validator functions themselves, but not for the actual job types. The problems with this approach include:

  1. It obscures the fact that when you're adding/modifying validator functions, you're also adding/modifying API behavior; e.g. you might change how a validator function raises exceptions, even if it's just revising an error message, without thinking about the fact that you're changing the behavior of the corresponding API responses, which is a breaking change.

  2. We're not testing the order in which validators are executed for a given job type. It might be important to call one validator before another (e.g. if the second validator is slower, so you only want it to run if the first one succeeds). I believe they run in the same order that they're listed in the job spec, but we don't test this for all of our job types.

  3. We're mocking the input data to the validator functions (job dict and list of CMR records), so our tests won't help us catch bugs that occur before the validator function is called (e.g. during CMR querying/filtering). See https://github.com/ASFHyP3/hyp3/issues/2737 for why this is important.

I think we should start including comprehensive integration tests for API job validation going forward.

Dominant language
Python
Stars
53
Forks
13
Avg merge
1d 2h
Merged PRs (30d)
20

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ASFHyP3/hyp3

All issues in ASFHyP3/hyp3

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.