HyP3 jobs with mapping overwhelm the batch queue

Open
  • Is something wrong?
#3,167 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Quiet
Tech stack
aws, python
Domain
backend, cloud

Research direction

Start by tracing the mapping step that submits LAVAs jobs to AWS Batch and inspect the related work in issue #3020. The issue suggests limiting concurrent submissions or adding backoff for throttling, but leaves the approach open; done means choosing and implementing a suitable rate limit strategy so mapped jobs avoid overwhelming the batch queue.

Written by the indexing model from the issue text.

Description

For LAVAs jobs, we're seeing too slow downs from AWS Batch submits because we don't adequetly handle the rate limits when submitting a bunch of jobs from a mapping step (when we fan out).

We should figure out how to handle this more appropriately, which might look like:

  • submitting fewer jobs at a time
  • catching slow down responses and backing off (exponential with jitter?)
  • rethinking this bit entirely (in conjunction with #3020 )
Dominant language
Python
Stars
53
Forks
13
Avg merge
1d 2h
Merged PRs (30d)
20

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ASFHyP3/hyp3

All issues in ASFHyP3/hyp3

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.