Slow cloud request/response tagging in Docker Lambda function with datadog-lambda

Open
#633 2 comments 0 reactions 1 assignee View on GitHub

@joeyzhao2018 is already working on this.

Since Jul 14, 2025.

Assessment

This issue has not been assessed yet.

Description

Hi, not sure if this is the right place or it should rather go to ddtrace or somewhere else:

Reproduction repo with CDK: https://github.com/maxfriedrich/datadog-cloud-payload-tagging-repro

The cloud request/response payload tagging feature causes a big slowdown for us for any DynamoDB query, even when "disabled" with the environment variable value "$.*" (0.xx seconds to > 10 seconds query), with the following setup:

  • Docker Lambda based on the Lambda Python 3.11 (or 3.13) base image
  • Installing the Datadog Lambda extension into the container
  • Installing datadog-lambda and ddtrace with into the container
  • Running the function with datadog_lambda.handler

(this function outgrew the zip size limit, this is why it's built like this -- is this a supported setup?)

We can work around it by setting the environment variable values to "", fully disabling the feature, not just redacting all fields.

The problem only occurs with a Docker based Lambda function, the handler from my reproduction is fine in the Python runtime and the Datadog extension and Lambda layers.

Output from my test script:

Testing slow function: arn:aws:lambda:eu-central-1:...:function:DdTaggingReproStack-Function1904AA941-QLjJ2nwy96AV
Execution time: 11.545 seconds
Execution time: 10.865 seconds
Execution time: 11.279 seconds
Execution time: 11.604 seconds
Execution time: 11.659 seconds

Testing fast function: arn:aws:lambda:eu-central-1:...:function:DdTaggingReproStack-Function28EA0674A-36P7xO9x9CbG
Execution time: 0.235 seconds
Execution time: 0.188 seconds
Execution time: 0.168 seconds
Execution time: 0.194 seconds
Execution time: 0.206 seconds

Do you have any ideas what could cause this? Feel free to move the issue if it belongs somewhere else. Thank you very much!

Dominant language
Python
Stars
102
Forks
52
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from DataDog/datadog-lambda-python

All issues in DataDog/datadog-lambda-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.