bug: OTEL_EXPORTER_OTLP_ENDPOINT in the environment breaks trigger deploy (depot build)

Abierto
  • ¿Algo no está bien?
#4,321 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

  • #4370 de @eeshsaxena — cerrado sin fusionar

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
45/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Estancado
Stack tecnológico
node.js, typescript
Área
devops

Línea de trabajo

Empieza por la invocación del proceso hijo de depot en packages/cli-v3/src/deploy/buildImage.ts, donde el objeto env que se pasa a depot() se fusiona con process.env mediante extendEnv de execa. Lee primero el PR #4370 cerrado y no mergeado para ver qué enfoque rechazaron los maintainers antes de proponer cómo eliminar o neutralizar OTEL_EXPORTER_OTLP_* para el hijo de depot manteniendo PATH/HOME intactos. Se considera terminado cuando: un deploy con OTEL_EXPORTER_OTLP_ENDPOINT establecido completa el build remoto de depot, y la ruta --local-build sigue funcionando.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Provide environment information
System:
  OS: macOS 15 (also reproduced on Linux, GitHub Actions ubuntu-24.04)
Binaries:
  Node: 22.x
  npm / pnpm / bun: any
trigger.dev CLI 4.5.4, @trigger.dev/sdk 4.5.4, @trigger.dev/build 4.5.4
@depot/cli 0.0.1-cli.2.80.0 (pinned by trigger.dev 4.5.4)
Describe the bug

In plain terms: OTEL_EXPORTER_OTLP_ENDPOINT is a normal, standard
environment variable that lots of projects already set — it tells OpenTelemetry
where to send traces. If that variable is set when you run trigger deploy, the
deploy fails while building the image. Remove the variable and the exact same
project deploys fine.

That's the bug: a value the user put in their own environment, for their own
unrelated tooling, breaks trigger.dev's build. A user-set environment variable
shouldn't be able to do that. The deploy should either ignore it or keep it out
of the build, not fall over on it. The error messages also point at "depot" and
look like an outage, so it's easy to lose a lot of time blaming the wrong thing
(we did) instead of realizing one of your own env vars is the cause.

Here's what actually happens under the hood.

If OTEL_EXPORTER_OTLP_ENDPOINT is set in the environment, trigger deploy
fails during the remote (depot) image build, before the build really starts.
The variable's value only changes which error you get:

  • a valid URL (e.g. http://localhost:4318) →
    Error: cannot merge resource due to conflicting Schema URL
  • any non-URL value (in our case a dotenvx encrypted:… blob) →
    failed to build resolver: passthrough: received empty target in Build()

Both surface as Error building image, with that line as the last log. Unset
the variable and the same project deploys fine on the same depot backend
seconds later. --local-build is unaffected.

We chased this for a while as flaky depot infrastructure (the empty target
error reads like a depot outage). It isn't — it's deterministic per-environment,
and it's the OTLP endpoint variable leaking into the depot build.

Root cause. The remote build runs the depot CLI via
packages/cli-v3/src/deploy/buildImage.ts:

const childProcess = depot(args, {
  cwd: options.cwd,
  env: {
    DEPOT_BUILD_ID: options.buildId,
    DEPOT_TOKEN: options.buildToken,
    DEPOT_PROJECT_ID: options.buildProjectId,
    DEPOT_NO_SUMMARY_LINK: "1",
    DEPOT_NO_UPDATE_NOTIFIER: "1",
  },
});

depot() (from @depot/cli) runs the binary through execa, whose default is
extendEnv: true. So that env object is merged onto the parent process.env
rather than replacing it — the depot CLI inherits every variable trigger deploy has, including OTEL_EXPORTER_OTLP_ENDPOINT. The depot CLI then reads it
for its own OTel export and dies during telemetry init (empty gRPC target for a
malformed value, resource Schema-URL conflict for a valid one) before it builds
anything.

The value gets into the environment easily. A common case is a .env file: the
CLI loads it with plain dotenv, so whatever is in it ends up in the
environment the depot child inherits. (Ours is encrypted with dotenvx, so the
value the CLI sees is the literal encrypted:… string — a non-URL — which is
why we hit the empty target variant rather than the Schema-URL one.) Setting
the variable directly in the shell does the same thing.

Reproduction repo

https://github.com/cloudbring/trigger-depot-otel-endpoint-repro

To reproduce

From the repo above (trivial single task, nothing project-specific):

npm install
npx [email protected] login
export TRIGGER_PROJECT_REF=proj_xxxxxxxx

# fails — .env sets OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
npm run deploy

# succeeds — same project, same code, variable removed
mv .env .env.off && npm run deploy

Or without the file:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 npx [email protected] deploy   # fails
npx [email protected] deploy                                                     # succeeds
Additional information

--local-build is a working workaround, and so is unsetting
OTEL_EXPORTER_OTLP_ENDPOINT for the deploy — the latter keeps you on remote
builds. The local path runs docker buildx (x("docker", ["buildx", …]) in the
same file) which inherits the same variable but tolerates it, so the hard
failure is specific to the depot CLI's telemetry init.

Since some ambient env (PATH, HOME, …) is needed for depot to run, dropping
extendEnv wholesale probably isn't the fix. Options that would work: strip or
neutralize the OTel exporter variables for the depot child (e.g. delete
OTEL_EXPORTER_OTLP_ENDPOINT / set OTEL_SDK_DISABLED=true in that env), or
have the depot CLI not fail the build when its own telemetry endpoint is bad.
Happy to open a PR for the CLI-side version if that's the direction you'd want.

Lenguaje dominante
TypeScript
Estrellas
16.5k
Forks
1.5k
Merge medio
1 d 15 h
PR fusionados (30 d)
17

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de triggerdotdev/trigger.dev

Todos los issues de triggerdotdev/trigger.dev

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.