503 Service Unavailable due to CMR Payload Too Large
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
Research direction
Start with apps/api/src/hyp3_api/validation.py, especially _get_cmr_metadata and validate_jobs, which the issue identifies as the CMR error and the source of the oversized query. Check how CMR reports a 413 response and how job validation gathers granules. Done means returning a descriptive error for this case and batching CMR queries to stay under its limit.
Written by the indexing model from the issue text.
Description
Jira: https://asfdaac.atlassian.net/browse/TOOL-3721
Note: The above link is accessible only to members of ASF.
When attempting to submit jobs like:
jobs = sdk.Batch()
for batch in sdk.util.chunk(prepared_jobs):
jobs += hyp3.submit_prepared_jobs(batch)
I kept getting these errors:
requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: https://hyp3-jhk-sandbox.asf.alaska.edu/jobs
hyp3_sdk.exceptions.ServiceUnavailableError: <Response [503]> Could not submit jobs due to a CMR error. Please try again later.
However, there was no CMR outage at the time. Looking at the API logs, _get_cmr_metadata is raising this error:
CMR search failed: 413 Client Error: Payload Too Large for url: https://cmr.earthdata.nasa.gov/search/granules.json
Since validate_jobs is gathering all the granules from all the jobs, the payload can exceed the CMR limit.
We should:
- Return a descriptive error in this case
- Batch queries to CMR to stay under the limit
- Dominant language
- Python
- Stars
- 53
- Forks
- 13
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 20
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from ASFHyP3/hyp3
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Most static tests fail when a date-like string is provided as an example value in a job spec YAMLOpenbug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
documentation
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Maintainers usually reply within 1 day
-
Jira Bug
Difficulty 1/5 1-3 hours Newbie friendliness 77/100
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
FinanceFlash/unvibecode#699 ·
Maintainers usually reply within 1 day
-
enhancement Website
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
OWASP/OWASP-Hunting#96 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
micropython/micropython#19942 ·
Maintainers usually reply within 1 day
-
A rebuild posts "not part of this build" success to the commit's failed `nix-build` gate and effectsPossibly taken @Padraic-O-Mhuiris claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
`epoch.Integer` raises a raw OverflowError for integers too large to convert to a floatPossibly taken @arindam77 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
pydantic/pydantic-extra-types#479 ·
Maintainers usually reply within 1 day