Add region to cloudwatch (and others?) log collection

Open
  • Is something wrong?
#685 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
aws, python
Domain
cloud, observability-sre

Research direction

Start with aws/logs_monitoring/parsing.py around line 503 and compare the event structure with the linked AWS CloudWatch Logs documentation. Reproduce the case using same-named log groups and streams in multiple regions, then determine whether the forwarder metadata can identify the source region. Done means the resulting log metadata exposes the requested region at aws.awslogs.region, with the behavior checked against the documented event context.

Written by the indexing model from the issue text.

Description

Describe what happened:
I have a log-forwarder configured on my account pulling cloudwatch logs. However it's difficult to determine which log stream it's coming from since the region is not included in any of the metadata in DD.

{
	"id": "..."
	"content": {
		"timestamp": "2023-08-07T14:52:03.221Z",
		"tags": [
			"forwardername:datadog-log-forwarder",
			"forwarder_version:3.73.0",
			"source:apigateway",
			"sourcecategory:aws",
			"forwarder_memorysize:1024",
			"datadog.submission_auth:private_api_key"
		],
		"host": "my-log-group",
		"service": "apigateway",
		"attributes": {
			"aws": {
				"awslogs": {
					"owner": "670180622055",
					"logGroup": "my-log-group",
					"logStream": "deadbeef.default-2023-08-07-14-52"
				},
				"invoked_function_arn": "arn:aws:lambda:us-east-1:13370000:function:datadog-log-forwarder",
				"function_version": "$LATEST"
			},
			"timestamp": 1691419923221
		}
	}
}

Describe what you expected:
I expected to have a region attribute at aws.awslogs.region so that I might uniquely identify from where the logs are being emitted.

Steps to reproduce the issue:
Use cloudwatch log streams in multiple regions with a forwarder in each region and cloudwatch log groups of the same name.

More information:
I looked at the code, and there aren't any samples but I believe the event record is documented here. It doesn't look like AWS provides anything useful at this level. So I'm hoping the event metadata might provide more context as to where the event came from. Since forwarders are configured in the specific region, perhaps it is sufficient to simply attach metadata regarding the current forwarders source. (I'm not 100% sure, since that would obscure cross-region log collection if it was being done.)

Workaround:
I am currently working around this by adding a custom tag to each of my cloudwatch log groups and setting DD_FETCH_LOG_GROUP_TAGS=true. But I believe this should be a built-in behavior if possible as multi-region deployments are a thing.

Thank you!

Dominant language
Python
Stars
351
Forks
397
Avg merge
7h 56m
Merged PRs (30d)
34

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from DataDog/datadog-serverless-functions

All issues in DataDog/datadog-serverless-functions

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.