Support fan-out jobs with arbitrary number of inputs

Open
  • Is something wrong?
#2,722 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
aws, python
Domain
backend, cloud

Research direction

Start with the referenced SRG_TIME_SERIES.yml job spec and the AWS Step Functions Map state documentation linked in the issue. Trace how fan-out inputs and state data are passed through the workflow, then assess Distributed mode and the listed quota and payload-size concerns. Done means fan-out/fan-in supports an arbitrary number of inputs within the relevant Step Functions quotas.

Written by the indexing model from the issue text.

Description

enhancement Jira Story

Jira: https://asfdaac.atlassian.net/browse/TOOL-3621

Note: The above link is accessible only to members of ASF.

There are some challenges to supporting fan-out/fan-in jobs with an arbitrary number of inputs (i.e. granules).

For one, throughput is limited by the maximum number of parallel iterations for the step function's Map state. The Map state defaults to Inline mode, which means it only accepts the input list as a JSON array and only runs 40 parallel iterations according to the docs. However, we can use Distributed mode, which allows the Map state to accept input as a CSV or JSON file in S3 (or an S3 object list) and runs up to 10,000 parallel iterations. For SRG_TIME_SERIES in particular, this would allow us to take full advantage of the G-instance vCPU quota in the LAVAS account. Also see:

There are also Step Functions service quotas that we may bump up against depending on the number of inputs we want to support; we've already addressed one of these in HyP3 v10.0.1, but there are other opportunities for reducing the size of the JSON data being passed between states, such as no longer passing around the job parameters (which include the full granules list) after they’re no longer needed, and writing the processing_times list (which includes one float value for each Batch job) directly to the database.

Dominant language
Python
Stars
53
Forks
13
Avg merge
1d 2h
Merged PRs (30d)
20

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ASFHyP3/hyp3

All issues in ASFHyP3/hyp3

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.