Duplicate product files sometimes exist for HyP3 jobs

Open
  • Is something wrong?
#2,755 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
aws, python
Domain
backend, cloud

Research direction

Start by tracing how HyP3 retries failed jobs and uploads product files to S3; the issue does not name specific files or tests. Reproduce or inspect a retry interrupted during upload, then determine how the job response and bucket contents should handle partial and retried uploads. Done means retries no longer leave duplicate product files listed for the same job.

Written by the indexing model from the issue text.

Description

bug

HyP3 product files often include a unique random hash to not clobber previously generated products using the same inputs.

HyP3 also utilizes the AWS SPOT market with retries for failed jobs.

Occasionally^, a job may fail due to being interrupted at the end of a job while uploading its files, and thus will end up with multiple duplicate product files for the same job upon retry.

Those duplicate files are listed in the job response and are viewable in the AWS S3 bucket (for deployments with public buckets).

For example, this job:
https://hyp3-cargill.asf.alaska.edu/jobs/134539d2-173f-4c9e-89cd-8f6cc11c2196

reports multiple product files, browse images, and thumbnail images. And if you list the bucket, you see:

$ aws  s3 ls s3://hyp3-cargill-contentbucket-1l3ubqacgyr1a/134539d2-173f-4c9e-89cd-8f6cc11c2196/
2025-05-07 06:49:07       1056 S1_195695_IW3_20220619_20220725_VV_INT40_870D.txt
2025-05-07 06:49:05   56399313 S1_195695_IW3_20220619_20220725_VV_INT40_870D.zip
2025-05-07 06:49:07      48897 S1_195695_IW3_20220619_20220725_VV_INT40_870D_conncomp.tif
2025-05-07 06:49:07    4289398 S1_195695_IW3_20220619_20220725_VV_INT40_870D_corr.tif
2025-05-07 06:49:07    1102674 S1_195695_IW3_20220619_20220725_VV_INT40_870D_dem.tif
2025-05-07 06:49:08    5347868 S1_195695_IW3_20220619_20220725_VV_INT40_870D_lon_rdr.tif
2025-05-07 06:49:06   10833599 S1_195695_IW3_20220619_20220725_VV_INT40_870D_los_rdr.tif
2025-05-07 06:49:06    2441332 S1_195695_IW3_20220619_20220725_VV_INT40_870D_lv_phi.tif
2025-05-07 06:49:08     448701 S1_195695_IW3_20220619_20220725_VV_INT40_870D_unw_phase.png
2025-05-07 06:49:07       1543 S1_195695_IW3_20220619_20220725_VV_INT40_870D_unw_phase_thumb.png
2025-05-07 06:49:06      27116 S1_195695_IW3_20220619_20220725_VV_INT40_870D_water_mask.tif
2025-05-07 06:49:07    4915198 S1_195695_IW3_20220619_20220725_VV_INT40_870D_wrapped_phase.tif
2025-05-07 07:00:13       1056 S1_195695_IW3_20220619_20220725_VV_INT40_D52A.txt
2025-05-07 07:00:11   56399311 S1_195695_IW3_20220619_20220725_VV_INT40_D52A.zip
2025-05-07 07:00:15      22454 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_README.md.txt
2025-05-07 07:00:13      48897 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_conncomp.tif
2025-05-07 07:00:13    4289398 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_corr.tif
2025-05-07 07:00:14    1102674 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_dem.tif
2025-05-07 07:00:14    4889137 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_lat_rdr.tif
2025-05-07 07:00:13    5347868 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_lon_rdr.tif
2025-05-07 07:00:15   10833599 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_los_rdr.tif
2025-05-07 07:00:14    2441332 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_lv_phi.tif
2025-05-07 07:00:14    3294233 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_lv_theta.tif
2025-05-07 07:00:15     448701 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_unw_phase.png
2025-05-07 07:00:14    4356826 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_unw_phase.tif
2025-05-07 07:00:14       1543 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_unw_phase_thumb.png
2025-05-07 07:00:14      27116 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_water_mask.tif
2025-05-07 07:00:15    4915198 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_wrapped_phase.tif
2025-05-07 07:00:13   16945191 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_wrapped_phase_rdr.tif

with the D52A product files being uploaded ~11 minutes after the partial upload of the 870D product files.

I have not quantified how often this happens, but I suspect rarely is a more appropriate descriptor.

Dominant language
Python
Stars
53
Forks
13
Avg merge
1d 2h
Merged PRs (30d)
20

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ASFHyP3/hyp3

All issues in ASFHyP3/hyp3

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.