bug: OTEL_EXPORTER_OTLP_ENDPOINT in the environment breaks trigger deploy (depot build)

Aperta
  • Qualcosa non va?
#4,321 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

  • #4370 di @eeshsaxena — chiusa senza merge

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
45/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Ferma
Stack tecnologico
node.js, typescript
Ambito
devops

Direzione di ricerca

Parti dall'invocazione del processo figlio depot in packages/cli-v3/src/deploy/buildImage.ts, dove l'oggetto env passato a depot() viene unito a process.env tramite extendEnv di execa. Leggi prima la PR #4370 chiusa e non mergiata per vedere quale approccio i maintainer hanno rifiutato prima di proporre come eliminare o neutralizzare OTEL_EXPORTER_OTLP_* per il figlio depot mantenendo PATH/HOME intatti. È considerato completato quando: un deploy con OTEL_EXPORTER_OTLP_ENDPOINT impostato completa il build remoto di depot, e il percorso --local-build funziona ancora.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Provide environment information
System:
  OS: macOS 15 (also reproduced on Linux, GitHub Actions ubuntu-24.04)
Binaries:
  Node: 22.x
  npm / pnpm / bun: any
trigger.dev CLI 4.5.4, @trigger.dev/sdk 4.5.4, @trigger.dev/build 4.5.4
@depot/cli 0.0.1-cli.2.80.0 (pinned by trigger.dev 4.5.4)
Describe the bug

In plain terms: OTEL_EXPORTER_OTLP_ENDPOINT is a normal, standard
environment variable that lots of projects already set — it tells OpenTelemetry
where to send traces. If that variable is set when you run trigger deploy, the
deploy fails while building the image. Remove the variable and the exact same
project deploys fine.

That's the bug: a value the user put in their own environment, for their own
unrelated tooling, breaks trigger.dev's build. A user-set environment variable
shouldn't be able to do that. The deploy should either ignore it or keep it out
of the build, not fall over on it. The error messages also point at "depot" and
look like an outage, so it's easy to lose a lot of time blaming the wrong thing
(we did) instead of realizing one of your own env vars is the cause.

Here's what actually happens under the hood.

If OTEL_EXPORTER_OTLP_ENDPOINT is set in the environment, trigger deploy
fails during the remote (depot) image build, before the build really starts.
The variable's value only changes which error you get:

  • a valid URL (e.g. http://localhost:4318) →
    Error: cannot merge resource due to conflicting Schema URL
  • any non-URL value (in our case a dotenvx encrypted:… blob) →
    failed to build resolver: passthrough: received empty target in Build()

Both surface as Error building image, with that line as the last log. Unset
the variable and the same project deploys fine on the same depot backend
seconds later. --local-build is unaffected.

We chased this for a while as flaky depot infrastructure (the empty target
error reads like a depot outage). It isn't — it's deterministic per-environment,
and it's the OTLP endpoint variable leaking into the depot build.

Root cause. The remote build runs the depot CLI via
packages/cli-v3/src/deploy/buildImage.ts:

const childProcess = depot(args, {
  cwd: options.cwd,
  env: {
    DEPOT_BUILD_ID: options.buildId,
    DEPOT_TOKEN: options.buildToken,
    DEPOT_PROJECT_ID: options.buildProjectId,
    DEPOT_NO_SUMMARY_LINK: "1",
    DEPOT_NO_UPDATE_NOTIFIER: "1",
  },
});

depot() (from @depot/cli) runs the binary through execa, whose default is
extendEnv: true. So that env object is merged onto the parent process.env
rather than replacing it — the depot CLI inherits every variable trigger deploy has, including OTEL_EXPORTER_OTLP_ENDPOINT. The depot CLI then reads it
for its own OTel export and dies during telemetry init (empty gRPC target for a
malformed value, resource Schema-URL conflict for a valid one) before it builds
anything.

The value gets into the environment easily. A common case is a .env file: the
CLI loads it with plain dotenv, so whatever is in it ends up in the
environment the depot child inherits. (Ours is encrypted with dotenvx, so the
value the CLI sees is the literal encrypted:… string — a non-URL — which is
why we hit the empty target variant rather than the Schema-URL one.) Setting
the variable directly in the shell does the same thing.

Reproduction repo

https://github.com/cloudbring/trigger-depot-otel-endpoint-repro

To reproduce

From the repo above (trivial single task, nothing project-specific):

npm install
npx [email protected] login
export TRIGGER_PROJECT_REF=proj_xxxxxxxx

# fails — .env sets OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
npm run deploy

# succeeds — same project, same code, variable removed
mv .env .env.off && npm run deploy

Or without the file:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 npx [email protected] deploy   # fails
npx [email protected] deploy                                                     # succeeds
Additional information

--local-build is a working workaround, and so is unsetting
OTEL_EXPORTER_OTLP_ENDPOINT for the deploy — the latter keeps you on remote
builds. The local path runs docker buildx (x("docker", ["buildx", …]) in the
same file) which inherits the same variable but tolerates it, so the hard
failure is specific to the depot CLI's telemetry init.

Since some ambient env (PATH, HOME, …) is needed for depot to run, dropping
extendEnv wholesale probably isn't the fix. Options that would work: strip or
neutralize the OTel exporter variables for the depot child (e.g. delete
OTEL_EXPORTER_OTLP_ENDPOINT / set OTEL_SDK_DISABLED=true in that env), or
have the depot CLI not fail the build when its own telemetry endpoint is bad.
Happy to open a PR for the CLI-side version if that's the direction you'd want.

Lingua principale
TypeScript
Stelle
16.5k
Fork
1.5k
Merge medio
1g 15h
PR unite (30g)
17

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di triggerdotdev/trigger.dev

Tutte le issue di triggerdotdev/trigger.dev

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.