With buildpack & auto sync a full re-build happens unpredictably

Tested using skaffold 1.17.2

1. Clone the Cloud Code Go hello-world k8s sample: https://github.com/GoogleCloudPlatform/cloud-code-samples/tree/master/golang/go-hello-world
2. Configure your build to use buildpacks:

build:
  tagPolicy:
    sha256: {}
  artifacts:
  - image: go-hello-world
    context: .
    buildpack:
      builder: gcr.io/buildpacks/builder:v1

3. Run skaffold dev
4. Make a change in main.go

Expected: Skaffolds auto-sync for buildpacks would automtically sync the change without performing a full container rebuild.

Actual: The full build happens.

Note: This only seems to happen unpredictably; in my tests, some changes sync, others trigger a full rebuild (the changes are identical each time - e.g. a simple fmt.Println string modification in this case).

Example output (the first couple changes did a full rebuild, the last change synced).