Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Go: Data race triggered by Encrypt/Decrypt (trips Go race detector)

Open
  • Is something wrong?
#902 5 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 5 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
42/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Quiet
Tech stack
go
Domain
cryptography, security

Research direction

Start by running TestHandler_Encrypt_And_Decrypt and TestHandler_DecryptAndVerify with Go's race detector. Trace the reported calls through EncryptDecryptHelpers, AwsEncryptionSdkOperations, ESDK, and the generated Dafny Base64 and Materials paths named in the stack trace. Done means the affected encryption and decryption tests pass under -race without data-race reports.

Written by the indexing model from the issue text.

Description

golang

While running tests using the AWS Encryption SDK for Go, we are seeing failed tests due to Go's race detector.

The relevant bits of the stack trace are:

WARNING: DATA RACE
Write at 0x00c000010d38 by goroutine 20:
  github.com/dafny-lang/DafnyRuntimeGo/v4/dafny.(*ArraySequence).IsString_set_()
      /home/runner/go/pkg/mod/github.com/dafny-lang/!dafny!runtime!go/[email protected]/dafny/dafnyFromDafny.go:937 +0x30
  github.com/dafny-lang/DafnyRuntimeGo/v4/dafny.(*CompanionStruct_Sequence_).SetString()
      /home/runner/go/pkg/mod/github.com/dafny-lang/!dafny!runtime!go/[email protected]/dafny/dafnyFromDafny.go:202 +0x18
  github.com/dafny-lang/DafnyRuntimeGo/v4/dafny.(*ArraySequence).SetString()
      /home/runner/go/pkg/mod/github.com/dafny-lang/!dafny!runtime!go/[email protected]/dafny/dafnyFromDafny.go:948 +0x2c
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).FromIndicesToChars()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/[email protected]/Base64/Base64.go:272 +0x48
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).EncodeUnpadded()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/[email protected]/Base64/Base64.go:290 +0x4c
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).Encode()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/[email protected]/Base64/Base64.go:361 +0x94
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/Materials.(*CompanionStruct_Default___).InitializeEncryptionMaterials.func2()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/[email protected]/Materials/Materials.go:224 +0x10a4
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/Materials.(*CompanionStruct_Default___).InitializeEncryptionMaterials()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/[email protected]/Materials/Materials.go:246 +0xb58
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/DefaultCMM.(*DefaultCMM).GetEncryptionMaterials_k()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/[email protected]/DefaultCMM/DefaultCMM.go:353 +0xc24
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/AwsCryptographyMaterialProvidersTypes.(*CompanionStruct_ICryptographicMaterialsManager_).GetEncryptionMaterials()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/[email protected]/AwsCryptographyMaterialProvidersTypes/AwsCryptographyMaterialProvidersTypes.go:3643 +0x50
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/DefaultCMM.(*DefaultCMM).GetEncryptionMaterials()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/[email protected]/DefaultCMM/DefaultCMM.go:281 +0x50
  github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/EncryptDecryptHelpers.(*CompanionStruct_Default___).GetEncryptionMaterials()
      /home/runner/go/pkg/mod/github.com/aws/aws-encryption-sdk/releases/go/[email protected]/EncryptDecryptHelpers/EncryptDecryptHelpers.go:691 +0x144
  github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/AwsEncryptionSdkOperations.(*CompanionStruct_Default___).Encrypt()
      /home/runner/go/pkg/mod/github.com/aws/aws-encryption-sdk/releases/go/[email protected]/AwsEncryptionSdkOperations/AwsEncryptionSdkOperations.go:380 +0x1184
  github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/ESDK.(*ESDKClient).Encrypt()
      /home/runner/go/pkg/mod/github.com/aws/aws-encryption-sdk/releases/go/[email protected]/ESDK/ESDK.go:435 +0x110
  github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/awscryptographyencryptionsdksmithygenerated.(*Client).Encrypt()
      /home/runner/go/pkg/mod/github.com/aws/aws-encryption-sdk/releases/go/[email protected]/awscryptographyencryptionsdksmithygenerated/api_client.go:38 +0x958
  <truncated>/encryption.(*awsEncryptionClient).Encrypt()
      ....
  <truncated>/encryption.TestHandler_Encrypt_And_Decrypt.func3()
      ....
  testing.tRunner()
      /home/runner/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1934 +0x164
  testing.(*T).Run.gowrap1()
      /home/runner/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1997 +0x3c

Previous write at 0x00c000010d38 by goroutine 19:
  github.com/dafny-lang/DafnyRuntimeGo/v4/dafny.(*ArraySequence).IsString_set_()
      /home/runner/go/pkg/mod/github.com/dafny-lang/!dafny!runtime!go/[email protected]/dafny/dafnyFromDafny.go:937 +0x30
  github.com/dafny-lang/DafnyRuntimeGo/v4/dafny.(*CompanionStruct_Sequence_).SetString()
      /home/runner/go/pkg/mod/github.com/dafny-lang/!dafny!runtime!go/[email protected]/dafny/dafnyFromDafny.go:202 +0x18
  github.com/dafny-lang/DafnyRuntimeGo/v4/dafny.(*ArraySequence).SetString()
      /home/runner/go/pkg/mod/github.com/dafny-lang/!dafny!runtime!go/[email protected]/dafny/dafnyFromDafny.go:948 +0x2c
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).FromIndicesToChars()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/[email protected]/Base64/Base64.go:272 +0x48
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).EncodeUnpadded()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/[email protected]/Base64/Base64.go:290 +0x4c
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).Encode()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/[email protected]/Base64/Base64.go:361 +0x94
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/Materials.(*CompanionStruct_Default___).InitializeEncryptionMaterials.func2()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/[email protected]/Materials/Materials.go:224 +0x10a4
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/Materials.(*CompanionStruct_Default___).InitializeEncryptionMaterials()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/[email protected]/Materials/Materials.go:246 +0xb58
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/DefaultCMM.(*DefaultCMM).GetEncryptionMaterials_k()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/[email protected]/DefaultCMM/DefaultCMM.go:353 +0xc24
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/AwsCryptographyMaterialProvidersTypes.(*CompanionStruct_ICryptographicMaterialsManager_).GetEncryptionMaterials()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/[email protected]/AwsCryptographyMaterialProvidersTypes/AwsCryptographyMaterialProvidersTypes.go:3643 +0x50
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/DefaultCMM.(*DefaultCMM).GetEncryptionMaterials()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/[email protected]/DefaultCMM/DefaultCMM.go:281 +0x50
  github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/EncryptDecryptHelpers.(*CompanionStruct_Default___).GetEncryptionMaterials()
      /home/runner/go/pkg/mod/github.com/aws/aws-encryption-sdk/releases/go/[email protected]/EncryptDecryptHelpers/EncryptDecryptHelpers.go:691 +0x144
  github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/AwsEncryptionSdkOperations.(*CompanionStruct_Default___).Encrypt()
      /home/runner/go/pkg/mod/github.com/aws/aws-encryption-sdk/releases/go/[email protected]/AwsEncryptionSdkOperations/AwsEncryptionSdkOperations.go:380 +0x1184
  github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/ESDK.(*ESDKClient).Encrypt()
      /home/runner/go/pkg/mod/github.com/aws/aws-encryption-sdk/releases/go/[email protected]/ESDK/ESDK.go:435 +0x110
  github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/awscryptographyencryptionsdksmithygenerated.(*Client).Encrypt()
      /home/runner/go/pkg/mod/github.com/aws/aws-encryption-sdk/releases/go/[email protected]/awscryptographyencryptionsdksmithygenerated/api_client.go:38 +0x958
  <truncated>/encryption.(*awsEncryptionClient).Encrypt()
      ....
  <truncated>/encryption.TestHandler_DecryptAndVerify.func1()
      ...
  testing.tRunner()
      /home/runner/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1934 +0x164
  testing.(*T).Run.gowrap1()
      /home/runner/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1997 +0x3c

Goroutine 20 (running) created at:
  testing.(*T).Run()
      /home/runner/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1997 +0x6e0
  <truncated>/encryption.TestHandler_Encrypt_And_Decrypt()
      ....
  testing.tRunner()
      /home/runner/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1934 +0x164
  testing.(*T).Run.gowrap1()
      /home/runner/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1997 +0x3c

Goroutine 19 (running) created at:
  testing.(*T).Run()
      /home/runner/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1997 +0x6e0
  <truncated>/encryption.TestHandler_DecryptAndVerify()
      ....
  testing.tRunner()
      /home/runner/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1934 +0x164
  testing.(*T).Run.gowrap1()
      /home/runner/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1997 +0x3c
==================
--- FAIL: TestHandler_Encrypt_And_Decrypt (0.00s)
    --- FAIL: TestHandler_Encrypt_And_Decrypt/Decrypts_ciphertext_produced_in_a_separate_Encrypt_call (0.05s)
        testing.go:1617: race detected during execution of test
    --- FAIL: TestHandler_Encrypt_And_Decrypt/Successfully_encrypts_and_decrypts (0.07s)
        testing.go:1617: race detected during execution of test
--- FAIL: TestHandler_DecryptAndVerify (0.00s)
    --- FAIL: TestHandler_DecryptAndVerify/Returns_plaintext_when_expected_context_matches (0.05s)
        testing.go:1617: race detected during execution of test
    --- FAIL: TestHandler_DecryptAndVerify/Empty_expected_context_behaves_like_Decrypt (0.05s)
        testing.go:1617: race detected during execution of test

It appears that the race is actually within Dafny, but filing the issue here as it's affecting the encryption SDK.

I used Claude to help investigate the root cause, the full detailed analysis provides much more detail.

AI Root Cause Analysis

Affected versions

  • github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk v0.3.0
  • github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl v0.3.0
  • github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library v0.3.0
  • github.com/dafny-lang/DafnyRuntimeGo/v4 v4.11.2
  • Go 1.25.x

Summary

The Go release of the AWS Encryption SDK is not goroutine-safe for concurrent use within a single process. Two goroutines that each call Client.Encrypt (or Decrypt) on independent ESDKClient instances — i.e. with no shared client state of their own — still race on a process-global ArraySequence inside the Dafny runtime. The Go -race detector flags this on every concurrent invocation.

Reproducer

package main

import (
    "context"
    "sync"
    "testing"

    mpl "github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/awscryptographymaterialproviderssmithygenerated"
    mpltypes "github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/awscryptographymaterialproviderssmithygeneratedtypes"
    esdkclient "github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/awscryptographyencryptionsdksmithygenerated"
    esdktypes "github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/awscryptographyencryptionsdksmithygeneratedtypes"
)

func TestConcurrentEncryptRaces(t *testing.T) {
    ctx := context.Background()
    var wg sync.WaitGroup
    for i := 0; i < 2; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            matProv, _ := mpl.NewClient(mpltypes.MaterialProvidersConfig{})
            keyring, _ := matProv.CreateRawAesKeyring(ctx, mpltypes.CreateRawAesKeyringInput{
                KeyName:      "k",
                KeyNamespace: "ns",
                WrappingKey:  []byte("superSecureKeySecureKey32bytes32"),
                WrappingAlg:  mpltypes.AesWrappingAlgAlgAes256GcmIv12Tag16,
            })
            c, _ := esdkclient.NewClient(esdktypes.AwsEncryptionSdkConfig{})
            _, _ = c.Encrypt(ctx, esdktypes.EncryptInput{
                Plaintext:         []byte("hello"),
                EncryptionContext: map[string]string{"k": "v"},
                Keyring:           keyring,
            })
        }()
    }
    wg.Wait()
}

Run with go test -race. The race detector reports the issue reliably.

Race trace (excerpt)

WARNING: DATA RACE
Write at 0x... by goroutine A:
  DafnyRuntimeGo/v4/dafny.(*ArraySequence).IsString_set_()
      dafny/dafnyFromDafny.go:937 +0x30
  DafnyRuntimeGo/v4/dafny.(*CompanionStruct_Sequence_).SetString()
      dafny/dafnyFromDafny.go:202 +0x18
  DafnyRuntimeGo/v4/dafny.(*ArraySequence).SetString()
      dafny/dafnyFromDafny.go:948 +0x2c
  smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).FromIndicesToChars()
      Base64/Base64.go:272 +0x48
  smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).EncodeUnpadded()
      Base64/Base64.go:290 +0x4c
  smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).Encode()
      Base64/Base64.go:361 +0x94
  mpl/Materials.(*CompanionStruct_Default___).InitializeEncryptionMaterials.func2()
      Materials/Materials.go:224 +0x10a4
  ...
  encryption-sdk/ESDK.(*ESDKClient).Encrypt()

Previous write at 0x... by goroutine B:
  [identical stack]

Root cause

Base64.FromIndicesToChars() returns a process-global ArraySequence (the base64 alphabet). Base64.Encode then calls SetString(true) on it, which dispatches to ArraySequence.IsString_set_ — an unsynchronized field write on the receiver (dafnyFromDafny.go:937). Because the alphabet sequence is shared, every concurrent Encrypt/Decrypt walks this code path and races on the same isString field.

The writes happen to be idempotent (always true), so on x86/ARM the field's final value is still correct in practice. But under the Go memory model, unsynchronized concurrent writes are undefined behavior, and -race flags them on every run, which makes the SDK unusable in any Go test suite that runs with -race (including the default for any project that values race-free CI).

Expected

Concurrent calls to Encrypt/Decrypt from independent ESDKClient instances complete without a race detector warning.

Suggested fix

Either:

  1. Make ArraySequence.IsString_set_ race-safe — e.g. back isString with sync/atomic or set it once via sync.Once since the value never transitions back.
  2. Avoid mutating the shared alphabet sequence in Base64.Encode; treat the alphabet as an immutable value and produce new sequences for the encoded output.

Option 2 is preferable because it eliminates a class of similar latent races in the broader Dafny stdlib and matches the immutable-sequence semantics Dafny otherwise advertises.

Workarounds

  • Run encryption tests serially (-parallel=1 or no t.Parallel()).
  • Add a process-wide mutex around every SDK call. This serializes all encryption in the process, which is unacceptable for servers that issue per-request KMS-backed encryption.

Neither is satisfying for a server runtime that does concurrent encryption per request.

Impact

Any Go service that calls Client.Encrypt/Decrypt from per-request goroutines (i.e. the typical pattern for encrypting PII on incoming HTTP requests) hits this race continuously. Production behavior is likely benign in practice because the racing write is idempotent, but the SDK cannot be cleanly validated under go test -race, and there is no upstream guidance on whether the SDK is intended to be goroutine-safe in this configuration.

Dominant language
Rust
Stars
68
Forks
29
Avg merge
4d 1h
Merged PRs (30d)
2

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from aws/aws-encryption-sdk

All issues in aws/aws-encryption-sdk

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.