s3_handler doesn't handle delete events

Open
#209 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
52/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
aws, python
Domain
backend, cloud

Research direction

Read steps/handlers/s3_handler.py, starting with handle and _extract_data, and inspect how the S3 event data identifies delete events or delete markers before get_object is called. Done means S3 delete notifications can be forwarded without attempting to retrieve the deleted object or raising NoSuchKey.

Written by the indexing model from the issue text.

Description

Expected Behavior

I want to forward logs from s3 delete event
s3_handler._extract_data shouldn't try to retreive an object if it has delete marker or even is of type delete

Actual Behavior

following method in s3_handler fails as it tries to access deleted object.
def _extract_data(self):
s3_client = self._get_s3_client()
response = s3_client.get_object(
Bucket=self.data_store.bucket, Key=self.data_store.key
)

body = response.get("Body")
self.data_store.data = body.read()

Steps to Reproduce the Problem

  1. Set up s3 bucket notification to trigger templated lambda upon object delete

Specifications

  • Datadog CloudFormation template version:

Stacktrace

[ERROR] NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.
Traceback (most recent call last):
File "/var/task/datadog_lambda/wrapper.py", line 239, in __call__
  """Executes when the wrapped function gets called"""
File "/var/task/ddtrace/contrib/internal/aws_lambda/patch.py", line 119, in __call__
  self.response = self.func(*args, **kwargs)
File "/var/task/datadog_lambda/wrapper.py", line 242, in __call__
  self.response = self.func(event, context, **kwargs)
File "/var/task/lambda_function.py", line 84, in datadog_forwarder
  parsed = parse(event, context, cache_layer)
File "/var/task/steps/parsing.py", line 56, in parse
  return normalize_events(events, metadata)
File "/var/task/steps/parsing.py", line 125, in normalize_events
  for event in events:
File "/var/task/steps/handlers/s3_handler.py", line 64, in handle
  self._extract_data()
File "/var/task/steps/handlers/s3_handler.py", line 118, in _extract_data
  response = s3_client.get_object(
File "/var/lang/lib/python3.12/site-packages/botocore/client.py", line 569, in _api_call
  return self._make_api_call(operation_name, kwargs)
File "/var/task/ddtrace/contrib/trace_utils.py", line 336, in wrapper
  return func(mod, pin, wrapped, instance, args, kwargs)
File "/var/task/ddtrace/contrib/internal/botocore/patch.py", line 200, in patched_api_call
  return patching_fn(
File "/var/task/ddtrace/contrib/internal/botocore/patch.py", line 260, in patched_api_call_fallback
  result = original_func(*args, **kwargs)
File "/var/lang/lib/python3.12/site-packages/botocore/client.py", line 1023, in _make_api_call
  raise error_class(parsed_response, operation_name)
Dominant language
Python
Stars
47
Forks
58
Avg merge
1d 18h
Merged PRs (30d)
8

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from DataDog/cloudformation-template

All issues in DataDog/cloudformation-template

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.