First file search in a project waits on a second walk of every folder to find symlinks: 551 ms of a 1.25 s first @-mention on Windows

オープン
#39,625 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

  • #39644 @juliusmarminge による — マージされずにクローズ

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
55/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
停滞
技術スタック
node.js, typescript

調査の方向性

Read apps/server/src/workspace/WorkspaceSearchIndex.ts, focusing on make, searchNames, and scanSymlinkEntries; the issue reports that symlink discovery rereads 4,703 indexed folders and blocks the first search. Start by tracing how fff builds its index and how symlink results are filtered after loadIgnoredPaths. Reproduce with the cold-search steps in the issue and compare the first-search trace; done means the search avoids or reduces the extra directory walk while still finding symlinks.

索引モデルが issue の本文から書いたものです。

説明

What is slow

The first file search in a project (the first @ mention in the composer, or the first file-picker query) waits until the server has built that project's search index. Most of that wait is spent finding symlinks. The server reads every folder the index already holds a second time, with one readdir per folder, and the search does not answer until that walk is done. The index is dropped after 15 minutes without a search (WORKSPACE_INDEX_IDLE_TTL), so a user pays this again whenever they come back to a project.

Measured

  • Windows Server 2025, 4 vCPU, 16 GB, NTFS, Git for Windows, Node 26.13.1. Linux comparison: Linux VM, ext4.
  • Commit f2d4df0367.
  • Seed: a checkout of this repository itself, 32,715 tracked files in 4,702 folders, 17 tracked symlinks.
  • Five runs per platform. Each run starts a fresh server on a fresh data dir (T3CODE_DEV_SEED=0), then sends one projects.searchEntries with query manager, limit 100. Times are server span durations from server.trace.ndjson.
Windows median (range) Linux median (range)
ws.rpc.projects.searchEntries (first search) 1,253 ms (1,228-1,425) 299 ms (244-431)
WorkspaceSearchIndex.scanSymlinkEntries 551 ms (502-623) 207 ms (172-251)
WorkspaceSearchIndex.loadIgnoredPaths (git ls-files --ignored) 661 ms (630-761) 59 ms (52-87)
WorkspaceSearchIndex.scanDirectory spans in the first search 4,703 4,703

Later searches in the same project take 18-32 ms on Windows. The symlink scan is the largest single part of the first search on Linux and about 44% of it on Windows.

How to measure it again

  1. Check out the commit, then run vp i.
  2. Start a server on a fresh data dir from apps/server:
    T3CODE_DEV_SEED=0 node src/bin.ts serve --base-dir <fresh-dir> --port 39880 --host 127.0.0.1
    
  3. Save this as apps/server/zz-search.ts and run node zz-search.ts http://127.0.0.1:39880 <fresh-dir> <path-to-a-large-repo>:
import * as NodeServices from "@effect/platform-node/NodeServices";
import * as NodeWS from "@effect/platform-node/NodeWS";
import { ORCHESTRATION_PROTOCOL_VERSION, WsRpcGroup } from "@t3tools/contracts";
import * as Effect from "effect/Effect";
import * as Layer from "effect/Layer";
import { RpcClient, RpcSerialization } from "effect/rpc";
import { Socket } from "effect/socket";
import { execFileSync } from "node:child_process";

const [origin, home, cwd] = process.argv.slice(2);
const token = execFileSync(process.execPath, ["src/bin.ts", "auth", "session", "issue", "--token-only", "--base-dir", home], { encoding: "utf8" }).trim();
const { ticket } = await (await fetch(`${origin}/api/auth/websocket-ticket`, { method: "POST", headers: { authorization: `Bearer ${token}` } })).json();
await Effect.runPromise(Effect.gen(function* () {
  const protocol = yield* Layer.build(RpcClient.layerProtocolSocket().pipe(
    Layer.provide(Socket.layerWebSocket(`${origin.replace(/^http/, "ws")}/ws?orchestrationProtocol=${ORCHESTRATION_PROTOCOL_VERSION}&wsTicket=${encodeURIComponent(ticket)}`)),
    Layer.provide(RpcSerialization.layerJson), Layer.provide(NodeWS.layerWebSocketConstructor)));
  const client = yield* RpcClient.make(WsRpcGroup).pipe(Effect.provideContext(protocol));
  for (let i = 0; i < 3; i++) {
    const t = Date.now();
    yield* client["projects.searchEntries"]({ cwd, query: "manager", limit: 100 });
    console.log(`search_ms=${Date.now() - t}`);
  }
}).pipe(Effect.scoped, Effect.provide(NodeServices.layer)));
process.exit(0);
  1. Wait about 10 s so the trace flushes. Stop the server, then read the spans of the first search from <fresh-dir>/userdata/logs/server.trace.ndjson:
import json, sys
sp = [json.loads(l) for l in open(sys.argv[1]) if l.strip()]
root = min((s for s in sp if s["name"] == "ws.rpc.projects.searchEntries"), key=lambda s: int(s["startTimeUnixNano"]))
t0, te = int(root["startTimeUnixNano"]), int(root["endTimeUnixNano"])
kids = [s for s in sp if s["traceId"] == root["traceId"] and t0 <= int(s["startTimeUnixNano"]) <= te]
for s in kids:
    if s["name"] in ("ws.rpc.projects.searchEntries", "WorkspaceSearchIndex.loadIgnoredPaths", "WorkspaceSearchIndex.scanSymlinkEntries"):
        print(f'{(int(s["startTimeUnixNano"]) - t0) / 1e6:8.1f} {s["durationMs"]:8.1f} {s["name"]}')
print("scanDirectory:", sum(s["name"] == "WorkspaceSearchIndex.scanDirectory" for s in kids))

Repeat steps 2-4 with a new data dir each time, so every run starts cold.

Where the time goes

The code is in apps/server/src/workspace/WorkspaceSearchIndex.ts.

  • make waits for loadIgnoredPaths() and waitForIndexReady.
  • searchNames then calls listSymlinkEntries. On the first read this starts scanSymlinkEntries and waits for it in full.
  • scanSymlinkEntries asks fff for every indexed folder (listIndexedDirectories). It then runs scanDirectory on each one, which is NodeFSP.readdir(..., { withFileTypes: true }), 16 at a time. That is 4,703 folder reads to find 17 links.

Timeline of one cold Windows first search (ms from the request):

   1.4    8.1  WorkspaceSearchIndex.createFinder
   9.7  621.9  WorkspaceSearchIndex.loadIgnoredPaths   (git ls-files --others --ignored)
 631.7    1.1  WorkspaceSearchIndex.waitForIndexReady
 635.8  540.0  WorkspaceSearchIndex.scanSymlinkEntries  (4,703 scanDirectory spans)
1193.1         ws.rpc.projects.searchEntries ends

Why it is avoidable

The tree has just been walked by fff. The server then walks every folder again only to find the few symlinks fff skips, and the first search waits for that walk. The walk also starts only after the git ignored-paths listing, although it does not need that listing until it filters its result.


Found by the automated Windows performance sweep on f2d4df0367.

主要言語
TypeScript
スター
24.8k
フォーク
6.4k
平均マージ
4時間 29分
マージ済み PR(30日)
488

環境構築

Codespaces で開く

このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

pingdotgg/t3code のほかの issue

pingdotgg/t3code の issue をすべて見る

似ている issue

TypeScript の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。