Validate granules using UMM_JSON metadata from CMR

Open
  • Is something wrong?
#2,737 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Refactor
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
backend

Research direction

Start with _get_cmr_metadata in apps/api/src/hyp3_api/validation.py and CMR_URL in apps/api/src/hyp3_api/__init__.py; compare the current granule JSON fields with their UMM_JSON equivalents. The issue says this work is blocked by issue #2740, so first check whether that integration test work is complete. Done means validators use the needed UMM_JSON metadata without regressions in existing validation behavior.

Written by the indexing model from the issue text.

Description

Jira Task

Jira: https://asfdaac.atlassian.net/browse/TOOL-3662

Note: The above link is accessible only to members of ASF.

The _get_cmr_metadata function in hyp3_api.validation currently queries https://cmr.earthdata.nasa.gov/search/granules.json as defined by CMR_URL here, and then further filters the fields included for each granule before returning the metadata.

Recently https://github.com/ASFHyP3/hyp3/issues/2739 motivated us to consider querying https://cmr.earthdata.nasa.gov/search/granules.umm_json instead, to get the UMM_JSON metadata, which includes more information than the JSON metadata.

@asjohnston-asf also points out here that using umm_json would simplify some of our other validators as well:

having access to the additional attributes simplifies check_same_relative_orbits via PATH_NUMBER and check_single_burst_pair via POLARIZATION

and the new check_rtc_static_coverage via BURST_ID_FULL

It shouldn't be too difficult to make the conversion, but unfortunately the fields available in umm_json are not a superset of the fields available in json, so it will require at least updating _get_cmr_metadata to filter for different fields, e.g. this is what currently gets returned from that function (as of v10.4.1):

    return [
        {
            'name': entry.get('producer_granule_id', entry.get('title')),
            'polygon': Polygon(_format_points(entry['polygons'][0][0])),
        }
        for entry in response.json()['feed']['entry']
    ]

so we'll have to find the equivalent of producer_granule_id, title, and polygons in the umm_json. But I'd probably lean toward removing the custom field filtering logic from _get_cmr_metadata and just providing the full umm_json CMR item for each granule to the validators, so that _get_cmr_metadata doesn't have to care about e.g. choosing between producer_granule_id and title for the granule name.

Even though it's probably a fairly straightforward refactor, I'm not confident in our ability to do this without introducing regression bugs for our existing job validators, especially since we don't have comprehensive integration tests that actually query CMR. So I would consider this issue blocked by https://github.com/ASFHyP3/hyp3/issues/2740.

Dominant language
Python
Stars
53
Forks
13
Avg merge
1d 2h
Merged PRs (30d)
20

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ASFHyP3/hyp3

All issues in ASFHyP3/hyp3

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.