A single transient metadata-server timeout permanently disables credentials for the life of the process

Comece rastreando os caminhos nomeados internal_lookup, post_update_existence e check_existence, incluindo TRANSIENT_EXCEPTIONS, e depois execute a reprodução metadata_latch_repro.rb. Está concluído quando uma falha transitória do servidor de metadados não impede permanentemente que uma sonda saudável posterior se recupere, enquanto o comportamento confirmado existente permanece intacto.

Environment

• google-cloud-env 2.4.0 (also confirmed in 2.2.1; the code is identical on main)
• Ruby 3.3, Linux
• Long-lived server processes (GitLab's puma and sidekiq workers) on a GCE VM

Summary

ComputeMetadata caches the outcome of its metadata-server probe for the life of the
process in both directions. Once a probe fails past the 60 s warmup window,
@existence is pinned to :no and never re-probed. Every subsequent Application
Default Credentials lookup in that process then fails with

RuntimeError: Your credentials were not found. To set up Application Default
Credentials for your environment, see
https://cloud.google.com/docs/authentication/external/set-up-adc

even though the metadata server was healthy again milliseconds later. The process
never recovers; only a restart clears it.

The defaults make this easy to hit: DEFAULT_OPEN_TIMEOUT = 0.1,
DEFAULT_REQUEST_TIMEOUT = 0.5. A 100 ms hiccup is enough.

Why we think this is a bug rather than intended caching

The failure is fed by the transport rescue in internal_lookup:

rescue *TRANSIENT_EXCEPTIONS
  post_update_existence false
  raise MetadataServerNotResponding

Exceptions the gem itself classifies as transient produce a permanent verdict.
post_update_existence writes @existence = :no, and check_existence then
short-circuits on it forever:

return current if [:no, :confirmed].include? @existence

Caching :confirmed for the life of the process is sound — a process does not move off
GCP. Caching :no is not: it promotes one timed-out request into a permanent statement
about the environment. That asymmetry is the defect.

The :no verdict also conflates two very different things: "the SMBIOS gate says we are
definitely not on GCP", which is genuinely permanent, and "one HTTP request timed out",
which is not.

Impact

Each worker latches independently, so the symptom presents as intermittent rather than
as an outage. In our case a poisoned puma process failed every object-storage operation
it handled while its siblings were fine — users saw HTTP 400 on merge request report
endpoints, and a page refresh appeared to fix it because a healthy process served the
retry. It went undiagnosed for about a week for exactly that reason.

Reproduction

Standalone, no GCP access needed — a fake metadata server on localhost with
GCE_METADATA_HOST pointed at it.

gem install google-cloud-env -v 2.4.0
ruby metadata_latch_repro.rb

#!/usr/bin/env ruby
# frozen_string_literal: true

# Reproduction: google-cloud-env caches a *negative* metadata-server verdict for
# the life of the process. One probe that times out past the 60 s warmup pins
# @existence = :no permanently, and every later Application Default Credentials
# lookup in that process fails, even though the metadata server is healthy again
# milliseconds later.
#
# No GCP access required. A fake metadata server runs on localhost and
# GCE_METADATA_HOST points at it.
#
#   gem install google-cloud-env -v 2.4.0
#   ruby metadata_latch_repro.rb
#
# K_SERVICE / K_REVISION / K_CONFIGURATION are set only to get past the SMBIOS
# hardware gate in gce_check, which would otherwise decide "definitely not on
# GCP" before any HTTP happens. On a real GCE VM or Cloud Run revision they are
# unnecessary; they are not part of the defect.
#
# warmup_time is set to 0 so the run takes seconds instead of a minute. It is
# equivalent to any process that has been alive longer than the default 60 s
# warmup, which is every long-lived worker in production.

require "socket"

# A metadata server that answers correctly, or stalls past the client's request
# timeout when @mode is :blip.
class FakeMetadataServer
  attr_accessor :mode

  def initialize
    @mode = :healthy
    @server = TCPServer.new "127.0.0.1", 0
    @requests = 0
    @thread = Thread.new { accept_loop }
  end

  def address
    "127.0.0.1:#{@server.addr[1]}"
  end

  attr_reader :requests

  def stop
    @thread.kill
    @server.close
  end

  private

  def accept_loop
    loop do
      socket = @server.accept
      Thread.new { handle socket }
    end
  rescue IOError
    nil
  end

  def handle socket
    socket.gets until socket.gets.to_s.strip.empty?
    @requests += 1
    # A blip: the connection opens, but no response arrives before the client's
    # 500 ms request timeout. This is what a momentarily overloaded metadata
    # server looks like from the client side.
    sleep 1.0 if @mode == :blip
    socket.print "HTTP/1.1 200 OK\r\nMetadata-Flavor: Google\r\nContent-Length: 2\r\n\r\nok"
  rescue StandardError
    nil
  ensure
    socket.close rescue nil
  end
end

server = FakeMetadataServer.new
ENV["GCE_METADATA_HOST"] = server.address
ENV["K_SERVICE"] = ENV["K_REVISION"] = ENV["K_CONFIGURATION"] = "repro"

require "google/cloud/env"
require "google/cloud/env/version"

def fresh_env
  env = Google::Cloud::Env.new
  env.compute_metadata.warmup_time = 0
  env
end

def report label, value
  puts format("    %-46s %s", label, value)
end

puts "google-cloud-env #{Google::Cloud::Env::VERSION}, metadata host #{ENV['GCE_METADATA_HOST']}"
puts

# Control. Proves the HTTP probe is what decides existence here, so the runs
# below are not being answered by the SMBIOS gate or by a cached verdict.
puts "[control] healthy server"
env = fresh_env
report "metadata?", env.metadata?
report "existence", env.compute_metadata.existence_immediate
puts

# The defect. A single blip while the server is briefly unresponsive, then the
# server is healthy for every subsequent call — and the process never recovers.
puts "[defect] one blip, then a healthy server forever"
env = fresh_env
server.mode = :blip
report "metadata? during the blip", env.metadata?
report "existence", env.compute_metadata.existence_immediate
server.mode = :healthy
before = server.requests
3.times { env.metadata? }
report "metadata? after the blip (server healthy)", env.metadata?
report "existence", env.compute_metadata.existence_immediate
report "HTTP requests made by those 3 calls", server.requests - before
puts

# The warmup boundary. The identical blip, on a process still inside its warmup
# window, leaves existence :unconfirmed and recovers on the next call. Only the
# warmup comparison in post_update_existence separates recovery from permanent
# failure, so what breaks a worker is being alive for more than 60 seconds.
puts "[control] the same blip, inside the warmup window"
env = Google::Cloud::Env.new
server.mode = :blip
report "metadata? during the blip", env.metadata?
report "existence", env.compute_metadata.existence_immediate
server.mode = :healthy
before = server.requests
report "metadata? after the blip", env.metadata?
report "existence", env.compute_metadata.existence_immediate
report "HTTP requests made by that call", server.requests - before
puts

# The asymmetry. A confirmed verdict cached forever is sound; the process is on
# GCP and will stay there. A negative verdict cached forever is not: it encodes
# a 100 ms network timeout as a permanent fact about the environment.
puts "[asymmetry] confirmed first, then the same blip"
env = fresh_env
env.metadata?
report "existence after a healthy probe", env.compute_metadata.existence_immediate
server.mode = :blip
env.metadata?
server.mode = :healthy
report "existence after the blip", env.compute_metadata.existence_immediate
puts

server.stop

puts <<~SUMMARY
  Expected: the [defect] run recovers once the server answers again.
  Actual:   existence is :no, check_existence short-circuits on it, and the
            three later calls make zero HTTP requests. In a real process every
            Application Default Credentials lookup from that point on raises
            "Your credentials were not found", until the process restarts.
SUMMARY

Output:

google-cloud-env 2.4.0, metadata host 127.0.0.1:39961

[control] healthy server
    metadata?                                      true
    existence                                      confirmed

[defect] one blip, then a healthy server forever
    metadata? during the blip                      false
    existence                                      no
    metadata? after the blip (server healthy)      false
    existence                                      no
    HTTP requests made by those 3 calls            0

[control] the same blip, inside the warmup window
    metadata? during the blip                      false
    existence                                      unconfirmed
    metadata? after the blip                       true
    existence                                      confirmed
    HTTP requests made by that call                1

[asymmetry] confirmed first, then the same blip
    existence after a healthy probe                confirmed
    existence after the blip                       confirmed

The two controls are what make the middle run meaningful:

• The first proves the HTTP probe, not the SMBIOS gate, is deciding existence here.
• The second runs the identical blip inside the warmup window: existence stays
  :unconfirmed and the next call recovers, having actually made a request. Only the
  warmup comparison in post_update_existence separates recovery from permanent
  failure — so what breaks a worker is simply having been alive for more than 60 s.

HTTP requests made by those 3 calls: 0 is the short-circuit: after latching, the
library stops talking to the metadata server entirely.

Suggested fix

We have no strong preference between these, and would rather defer to you:

1. Do not latch :no from a transport failure at all. Reserve :no for conclusive
   negatives (the SMBIOS/environment gate, or a server that answers with the wrong
   flavor), and leave a timed-out probe at :unconfirmed so the next lookup retries.
2. Give the :no verdict an expiry, so check_existence re-probes after some interval
   rather than short-circuiting forever.

Either would have prevented this. (1) seems closer to the existing intent, given that
the failing path is already named TRANSIENT_EXCEPTIONS.

Workaround, for anyone who finds this first

We install a Rails initializer that calls reset_existence! and then
ensure_existence(timeout: 10.0) at boot. The reset matters: Env.new runs at file
scope during Bundler.require, so the warmup clock is already stamped by the time
initializers run. Since :confirmed is cached just as permanently, confirming once at
boot makes the :no branch unreachable for the life of the process. That is a
workaround for the asymmetry, not a fix for it.