bug: OTEL_EXPORTER_OTLP_ENDPOINT in the environment breaks trigger deploy (depot build)

オープン
  • 何か問題がありますか？
#4,321 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

  • #4370 @eeshsaxena による — マージされずにクローズ

評価

難易度
3/5
見積もり時間
1〜2日
初心者へのやさしさ
45/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
停滞
技術スタック
node.js, typescript
領域
devops

調査の方向性

packages/cli-v3/src/deploy/buildImage.ts における depot の子プロセス呼び出しから始める。ここでは depot() に渡される env オブジェクトが execa の extendEnv 経由で process.env にマージされる。提案する前に、maintainer がどのアプローチを却下したかを見るため、まずクローズ済み・未マージの PR #4370 を読むこと。PATH/HOME を保ったまま、depot の子プロセスに対して OTEL_EXPORTER_OTLP_* をどうやって除去または無効化するかを提案する。完成の定義: OTEL_EXPORTER_OTLP_ENDPOINT を設定した deploy がリモートの depot ビルドを完了し、--local-build パスも引き続き動作する。

索引モデルが issue の本文から書いたものです。

説明

Provide environment information
System:
  OS: macOS 15 (also reproduced on Linux, GitHub Actions ubuntu-24.04)
Binaries:
  Node: 22.x
  npm / pnpm / bun: any
trigger.dev CLI 4.5.4, @trigger.dev/sdk 4.5.4, @trigger.dev/build 4.5.4
@depot/cli 0.0.1-cli.2.80.0 (pinned by trigger.dev 4.5.4)
Describe the bug

In plain terms: OTEL_EXPORTER_OTLP_ENDPOINT is a normal, standard
environment variable that lots of projects already set — it tells OpenTelemetry
where to send traces. If that variable is set when you run trigger deploy, the
deploy fails while building the image. Remove the variable and the exact same
project deploys fine.

That's the bug: a value the user put in their own environment, for their own
unrelated tooling, breaks trigger.dev's build. A user-set environment variable
shouldn't be able to do that. The deploy should either ignore it or keep it out
of the build, not fall over on it. The error messages also point at "depot" and
look like an outage, so it's easy to lose a lot of time blaming the wrong thing
(we did) instead of realizing one of your own env vars is the cause.

Here's what actually happens under the hood.

If OTEL_EXPORTER_OTLP_ENDPOINT is set in the environment, trigger deploy
fails during the remote (depot) image build, before the build really starts.
The variable's value only changes which error you get:

  • a valid URL (e.g. http://localhost:4318) →
    Error: cannot merge resource due to conflicting Schema URL
  • any non-URL value (in our case a dotenvx encrypted:… blob) →
    failed to build resolver: passthrough: received empty target in Build()

Both surface as Error building image, with that line as the last log. Unset
the variable and the same project deploys fine on the same depot backend
seconds later. --local-build is unaffected.

We chased this for a while as flaky depot infrastructure (the empty target
error reads like a depot outage). It isn't — it's deterministic per-environment,
and it's the OTLP endpoint variable leaking into the depot build.

Root cause. The remote build runs the depot CLI via
packages/cli-v3/src/deploy/buildImage.ts:

const childProcess = depot(args, {
  cwd: options.cwd,
  env: {
    DEPOT_BUILD_ID: options.buildId,
    DEPOT_TOKEN: options.buildToken,
    DEPOT_PROJECT_ID: options.buildProjectId,
    DEPOT_NO_SUMMARY_LINK: "1",
    DEPOT_NO_UPDATE_NOTIFIER: "1",
  },
});

depot() (from @depot/cli) runs the binary through execa, whose default is
extendEnv: true. So that env object is merged onto the parent process.env
rather than replacing it — the depot CLI inherits every variable trigger deploy has, including OTEL_EXPORTER_OTLP_ENDPOINT. The depot CLI then reads it
for its own OTel export and dies during telemetry init (empty gRPC target for a
malformed value, resource Schema-URL conflict for a valid one) before it builds
anything.

The value gets into the environment easily. A common case is a .env file: the
CLI loads it with plain dotenv, so whatever is in it ends up in the
environment the depot child inherits. (Ours is encrypted with dotenvx, so the
value the CLI sees is the literal encrypted:… string — a non-URL — which is
why we hit the empty target variant rather than the Schema-URL one.) Setting
the variable directly in the shell does the same thing.

Reproduction repo

https://github.com/cloudbring/trigger-depot-otel-endpoint-repro

To reproduce

From the repo above (trivial single task, nothing project-specific):

npm install
npx [email protected] login
export TRIGGER_PROJECT_REF=proj_xxxxxxxx

# fails — .env sets OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
npm run deploy

# succeeds — same project, same code, variable removed
mv .env .env.off && npm run deploy

Or without the file:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 npx [email protected] deploy   # fails
npx [email protected] deploy                                                     # succeeds
Additional information

--local-build is a working workaround, and so is unsetting
OTEL_EXPORTER_OTLP_ENDPOINT for the deploy — the latter keeps you on remote
builds. The local path runs docker buildx (x("docker", ["buildx", …]) in the
same file) which inherits the same variable but tolerates it, so the hard
failure is specific to the depot CLI's telemetry init.

Since some ambient env (PATH, HOME, …) is needed for depot to run, dropping
extendEnv wholesale probably isn't the fix. Options that would work: strip or
neutralize the OTel exporter variables for the depot child (e.g. delete
OTEL_EXPORTER_OTLP_ENDPOINT / set OTEL_SDK_DISABLED=true in that env), or
have the depot CLI not fail the build when its own telemetry endpoint is bad.
Happy to open a PR for the CLI-side version if that's the direction you'd want.

主要言語
TypeScript
スター
16.5k
フォーク
1.5k
平均マージ
1日 15時間
マージ済み PR（30日）
17

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

triggerdotdev/trigger.dev のほかの issue

triggerdotdev/trigger.dev の issue をすべて見る

似ている issue

TypeScript の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。