Parallelization Factor for MSK Event Source Mapping

Open
#2 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
20/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
aws
Domain
cloud

Research direction

No repository files, tests, or implementation entry points are mentioned. Start by reviewing the AWS Lambda MSK event-source documentation and the repository structure to determine whether this is implementable here; done would require a confirmed scope and an agreed way to support or document parallel processing.

Written by the indexing model from the issue text.

Description

Hi,

Is there a way to set concurrency of the lambda function so that multiple batches can be processed simultaneouly?

From the aws lambda cli docs I see there is a --parallelization-factor which I think is supported for Kinesis streams, but not MSK?

Are there any plans to support this for MSK?

If I understood the MSK event source correctly, it polls a target MSK topic across partitions and invokes a lambda function with the obtained batch synchronously? So in order to poll for another batch, the invoked lambda function must complete execution before a subsequent poll can begin?

Therefore, the throughput that can be handled is a function of:

  • The number of topic partitions (more partitions allows for higher throughput)
  • The batch size of the MSK event source (10,000 is currently the max)
  • The execution time of the lambda function.

For example, if the execution takes 100ms vs. 10secs this makes a difference to the consumer lag in high-throughput topics?

Many thanks,

David

Dominant language
Java
Stars
11
Forks
10
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from aws-samples/integration-sample-lambda-msk

All issues in aws-samples/integration-sample-lambda-msk

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.