GoogleCloudPlatform/terraform-splunk-log-export

replaying deadletter with replay udf and includePubsubMessage=true doesn't work

Open

#38 opened on Mar 3, 2023

View on GitHub
 (2 comments) (0 reactions) (0 assignees)HCL (32 forks)auto 404
bugdocumentationgood first issue

Repository metrics

Stars
 (46 stars)
PR merge metrics
 (PR metrics pending)

Description

The default configured behavior of the log export pipeline is to force includePubsubMessage to true[1].

When used in combination with sample Dataflow UDF message replay Javascript[2], replayed messages from the deadletter subscription won't be handled correctly. As messages enter the original/main log subscription, they will be re-wrapped in the data key again. You will end up with strange logs like this in Splunk:

{
  "data": {
    "time": 1677812444044,
    "event": "{\"data\":{\"insertId\":\"sv1hmye6bz6p\",\"logName\":\"projects/redacted-project/logs/[cloudaudit.googleapis.com](http://cloudaudit.googleapis.com/)%2Fdata_access\",\"protoPayload\":{\"@type\":\"[type.googleapis.com/google.cloud.audit.AuditLog\](http://type.googleapis.com/google.cloud.audit.AuditLog%5C)",\"authenticationInfo\":{\"principalEmail\":\"[redacteduser-sfx-scraper@redacted-project.iam.gserviceaccount.com](mailto:redacteduser-sfx-scraper@redacted-project.iam.gserviceaccount.com)\",\"principalSubject\":\"[serviceAccount:redacteduser-sfx-scraper@redacted-project.iam.gserviceaccount.com](mailto:serviceAccount%3Aredacteduser-sfx-scraper@redacted-project.iam.gserviceaccount.com)\",\"serviceAccountKeyName\":\"//[iam.googleapis.com/projects/redacted-project/serviceAccounts/redacteduser-sfx-scraper@redacted-project.iam.gserviceaccount.com/keys/a49e3382174e3775dec6fd4e5b593dbfed8c1ecc\](http://iam.googleapis.com/projects/redacted-project/serviceAccounts/redacteduser-sfx-scraper@redacted-project.iam.gserviceaccount.com/keys/a49e3382174e3775dec6fd4e5b593dbfed8c1ecc%5C)"},\"authorizationInfo\":[{\"granted\":true,\"permission\":\"monitoring.timeSeries.list\",\"resource\":\"50701599922\",\"resourceAttributes\":{}}],\"methodName\":\"google.monitoring.v3.MetricService.ListTimeSeries\",\"request\":{\"@type\":\"[type.googleapis.com/google.monitoring.v3.ListTimeSeriesRequest\](http://type.googleapis.com/google.monitoring.v3.ListTimeSeriesRequest%5C)",\"filter\":\"metric.type = \\\"[kubernetes.io/node/cpu/allocatable_utilization\\\](http://kubernetes.io/node/cpu/allocatable_utilization%5C%5C%5C)"\",\"name\":\"projects/redacted-project\",\"pageSize\":10000},\"requestMetadata\":{\"callerIp\":\"44.230.82.104\",\"callerSuppliedUserAgent\":\"grpc-java-netty/1.51.1,gzip(gfe)\",\"destinationAttributes\":{},\"requestAttributes\":{\"auth\":{},\"time\":\"2023-03-03T03:00:44.050942344Z\"}},\"resourceName\":\"projects/redacted-project\",\"serviceName\":\"[monitoring.googleapis.com](http://monitoring.googleapis.com/)\"},\"receiveTimestamp\":\"2023-03-03T03:00:44.174642381Z\",\"resource\":{\"labels\":{\"method\":\"google.monitoring.v3.MetricService.ListTimeSeries\",\"project_id\":\"redacted-project\",\"service\":\"[monitoring.googleapis.com](http://monitoring.googleapis.com/)\"},\"type\":\"audited_resource\"},\"severity\":\"INFO\",\"timestamp\":\"2023-03-03T03:00:44.044614574Z\"},\"attributes\":{\"[logging.googleapis.com/timestamp\](http://logging.googleapis.com/timestamp%5C)":\"2023-03-03T03:00:44.044614574Z\"},\"messageId\":\"7058887762594632\",\"delivery_attempt\":1}"
  },
  "attributes": {
    "errorMessage": "Splunk write status code: 403",
    "timestamp": "2023-03-03 03:00:34.000000"
  },
  "messageId": "7058953036869866",
  "delivery_attempt": 1
}

[1] - https://github.com/GoogleCloudPlatform/terraform-splunk-log-export/blob/main/main.tf#L85 [2] - https://storage.googleapis.com/splk-public/js/dataflow_udf_messages_replay.js

Contributor guide