Permission errors when accessing output files in S3

Open
  • Is something wrong?
#1,580 5 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
aws, python
Domain
backend, cloud

Research direction

Start by reviewing how completed job output files are represented by the API and why their S3 bucket and key are still returned after the migration described in the issue. Check the API’s file metadata and access configuration entry points; the report provides no filenames or tests to start from. Done means users can access output files through a supported path without the reported permission errors.

Written by the indexing model from the issue text.

Description

bug Jira Bug

Jira: https://asfdaac.atlassian.net/browse/TOOL-2040

Completed HyP3 jobs include an AWS S3 Bucket and Key for their output files, but attempting to access those files using the AWS CLI or the boto3 python library results in AccessDenied or Forbidden errors. For example:

>>> import hyp3_sdk
>>> hyp3 = hyp3_sdk.hyp3()
>>> job = hyp3.get_job_by_id('62db0f98-7c7e-42bf-ad2e-30840abad4ed')
>>> job.files[0]['s3']
{'bucket': 'hyp3-edc-prod-contentbucket-1fv14ed36ifj6', 'key': '62db0f98-7c7e-42bf-ad2e-30840abad4ed/S1AA_20150504T120217_20150621T120220_VVP048_INT80_G_ueF_ABE0.zip'}

>>> import boto3
>>> s3 = boto3.client('s3')
>>> s3.download_file(job.files[0]['s3']['bucket'], job.files[0]['s3']['key'], 'S1AA_20150504T120217_20150621T120220_VVP048_INT80_G_ueF_ABE0.zip')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/asjohnston/mambaforge/lib/python3.10/site-packages/boto3/s3/inject.py", line 190, in download_file
    return transfer.download_file(
  File "/home/asjohnston/mambaforge/lib/python3.10/site-packages/boto3/s3/transfer.py", line 326, in download_file
    future.result()
  File "/home/asjohnston/mambaforge/lib/python3.10/site-packages/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
  File "/home/asjohnston/mambaforge/lib/python3.10/site-packages/s3transfer/futures.py", line 266, in result
    raise self._exception
  File "/home/asjohnston/mambaforge/lib/python3.10/site-packages/s3transfer/tasks.py", line 269, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/home/asjohnston/mambaforge/lib/python3.10/site-packages/s3transfer/download.py", line 354, in _submit
    response = client.head_object(
  File "/home/asjohnston/mambaforge/lib/python3.10/site-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/asjohnston/mambaforge/lib/python3.10/site-packages/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

Accessing these files directly in S3 was possible until Jun 2022, when the https://hyp3-api.asf.alaska.edu/ API was migrated to a NASA-managed AWS platform where security controls prevent direct public access to files in S3. The bucket and key are still included in the description of each output file, but attempts to use boto3 or the AWS CLI now fail with permissions errors.

Dominant language
Python
Stars
53
Forks
13
Avg merge
1d 2h
Merged PRs (30d)
20

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ASFHyP3/hyp3

All issues in ASFHyP3/hyp3

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.