[BUG] Incorrect unmarshalling self-hosted Kafka event
@bmoffatt is already working on this.
Since Dec 20, 2022.
Assessment
This issue has not been assessed yet.
Description
We have a lambda function which is hooked up and listens to both MSK and self-hosted Kafka instances to process incoming messages. While in case is MSK the event source mapping works fine, for self-hosted event source mapping SDK panics with error as follow:
{
"errorMessage": "json: cannot unmarshal number -23 into Go struct field KafkaRecord.records.headers of type uint8",
"errorType": "UnmarshalTypeError"
}
This seems like some type mismatch (uint8) so it can't properly unmarshal negative numbers coming in the lambda event payload.
We had opened AWS support case and worked with them to some degree but unfortunately they couldn't provide us much context so we did debugging ourselves.
The issue seems to be present in the versions 1.26+ of the Lambda SDK, and downgrading to 1.25 or below doesn't cause this unmarshal error to appear. The main different is the presence of the headers field in the struct introduced through https://github.com/aws/aws-lambda-go/pull/385/files#diff-cf0e5ac9714618f1821f8694da87d7af4579e0eafed210e8dbdcc5977bf5a69eR20. This might also be related to some previous issue https://github.com/aws/aws-lambda-go/issues/390.
As mentioned above this only applies to self-hosted event mapping, not MSK events.
Environment:
- Go:
1.17 - AWS LAmbda SDK Go:
1.26,1.27,1.28 - Kafka version (self-hosted):
2.7.0
- Dominant language
- Go
- Stars
- 3.8k
- Forks
- 578
- Avg merge
- 16h 56m
- Merged PRs (30d)
- 2
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from aws/aws-lambda-go
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
aws/aws-lambda-go#634 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
aws/aws-lambda-go#597 · 3 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
aws/aws-lambda-go#556 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
aws/aws-lambda-go#610 · 1 comment ·
-
Difficulty 3/5 1-2 days Newbie friendliness 62/100
aws/aws-lambda-go#598 ·
All issues in aws/aws-lambda-go
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 84/100
-
enhancement needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
kind/cleanup
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
kubernetes-sigs/kueue#15947 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
sympozium-ai/sympozium#627 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100