replaying deadletter with replay udf and includePubsubMessage=true doesn't work
#38 opened on Mar 3, 2023
Repository metrics
- Stars
- (46 stars)
- PR merge metrics
- (PR metrics pending)
Description
The default configured behavior of the log export pipeline is to force includePubsubMessage to true[1].
When used in combination with sample Dataflow UDF message replay Javascript[2], replayed messages from the deadletter subscription won't be handled correctly. As messages enter the original/main log subscription, they will be re-wrapped in the data key again. You will end up with strange logs like this in Splunk:
{
"data": {
"time": 1677812444044,
"event": "{\"data\":{\"insertId\":\"sv1hmye6bz6p\",\"logName\":\"projects/redacted-project/logs/[cloudaudit.googleapis.com](http://cloudaudit.googleapis.com/)%2Fdata_access\",\"protoPayload\":{\"@type\":\"[type.googleapis.com/google.cloud.audit.AuditLog\](http://type.googleapis.com/google.cloud.audit.AuditLog%5C)",\"authenticationInfo\":{\"principalEmail\":\"[redacteduser-sfx-scraper@redacted-project.iam.gserviceaccount.com](mailto:redacteduser-sfx-scraper@redacted-project.iam.gserviceaccount.com)\",\"principalSubject\":\"[serviceAccount:redacteduser-sfx-scraper@redacted-project.iam.gserviceaccount.com](mailto:serviceAccount%3Aredacteduser-sfx-scraper@redacted-project.iam.gserviceaccount.com)\",\"serviceAccountKeyName\":\"//[iam.googleapis.com/projects/redacted-project/serviceAccounts/redacteduser-sfx-scraper@redacted-project.iam.gserviceaccount.com/keys/a49e3382174e3775dec6fd4e5b593dbfed8c1ecc\](http://iam.googleapis.com/projects/redacted-project/serviceAccounts/redacteduser-sfx-scraper@redacted-project.iam.gserviceaccount.com/keys/a49e3382174e3775dec6fd4e5b593dbfed8c1ecc%5C)"},\"authorizationInfo\":[{\"granted\":true,\"permission\":\"monitoring.timeSeries.list\",\"resource\":\"50701599922\",\"resourceAttributes\":{}}],\"methodName\":\"google.monitoring.v3.MetricService.ListTimeSeries\",\"request\":{\"@type\":\"[type.googleapis.com/google.monitoring.v3.ListTimeSeriesRequest\](http://type.googleapis.com/google.monitoring.v3.ListTimeSeriesRequest%5C)",\"filter\":\"metric.type = \\\"[kubernetes.io/node/cpu/allocatable_utilization\\\](http://kubernetes.io/node/cpu/allocatable_utilization%5C%5C%5C)"\",\"name\":\"projects/redacted-project\",\"pageSize\":10000},\"requestMetadata\":{\"callerIp\":\"44.230.82.104\",\"callerSuppliedUserAgent\":\"grpc-java-netty/1.51.1,gzip(gfe)\",\"destinationAttributes\":{},\"requestAttributes\":{\"auth\":{},\"time\":\"2023-03-03T03:00:44.050942344Z\"}},\"resourceName\":\"projects/redacted-project\",\"serviceName\":\"[monitoring.googleapis.com](http://monitoring.googleapis.com/)\"},\"receiveTimestamp\":\"2023-03-03T03:00:44.174642381Z\",\"resource\":{\"labels\":{\"method\":\"google.monitoring.v3.MetricService.ListTimeSeries\",\"project_id\":\"redacted-project\",\"service\":\"[monitoring.googleapis.com](http://monitoring.googleapis.com/)\"},\"type\":\"audited_resource\"},\"severity\":\"INFO\",\"timestamp\":\"2023-03-03T03:00:44.044614574Z\"},\"attributes\":{\"[logging.googleapis.com/timestamp\](http://logging.googleapis.com/timestamp%5C)":\"2023-03-03T03:00:44.044614574Z\"},\"messageId\":\"7058887762594632\",\"delivery_attempt\":1}"
},
"attributes": {
"errorMessage": "Splunk write status code: 403",
"timestamp": "2023-03-03 03:00:34.000000"
},
"messageId": "7058953036869866",
"delivery_attempt": 1
}
[1] - https://github.com/GoogleCloudPlatform/terraform-splunk-log-export/blob/main/main.tf#L85 [2] - https://storage.googleapis.com/splk-public/js/dataflow_udf_messages_replay.js