[Bug] LanceDbIndex interpolates paths into LanceDB SQL predicates — apostrophes in file/workspace paths break index deletes (stale chunks stay retrievable) and retrieval

Read core/indexing/LanceDbIndex.ts, focusing on the update() delete loop and _retrieveForTag; run the standalone vectordb@0.4.20 reproduction described in the issue. Ensure paths containing apostrophes work for deletion and retrieval without breaking ordinary predicates or leaving stale chunks retrievable.

Description

LanceDbIndex builds its LanceDB SQL predicates by string interpolation of on-disk paths and workspace directories, and a single quote is a legal character in both:

• delete path (core/indexing/LanceDbIndex.ts, update()):

  await lanceTable.delete(
    `cachekey = '${cacheKey}' AND path = '${path}'`,
  );
  

• retrieval path (_retrieveForTag):

  query = query.where(`path LIKE '${directory}%'`).limit(300);
  

A path or workspace directory containing an apostrophe (don't.py, it's a test/…) produces an unparseable predicate and LanceDB throws Unterminated string literal. Consequences:

1. Deleted/renamed files stay retrievable. compute/insert paths use parameterized sqlite and Arrow row adds (no interpolation), so a file with a quote in its name indexes fine. When the file is later deleted or renamed, update() reaches the toDel loop (removeTag + del) and the interpolated delete throws. CodebaseIndexer catches per batch, logs a warning, and moves on — markComplete never runs for that batch, so the tag-catalog rows survive and every subsequent sync retries the same throw. The stale chunks remain in that tag's LanceDB table and continue to surface in @Codebase retrieval even though the source file is gone.
2. Batch aborts. Every other file sharing the failing file's filesPerBatch batch also never completes and is re-processed on every sync.
3. Workspace directories with an apostrophe break retrieval. _retrieveForTag interpolates the directory into path LIKE '…%', so retrieval throws for the whole workspace.

This was found in a correctness study of derived-representation lifecycle handling (what happens to index entries when their source is revoked).

Environment

• continue pinned revision 5522c6f44ca0ac3528b37244818fbfa39b5af470 (2026-09), core/
• vectordb@0.4.20 (the version pinned in core/package.json), Node 24, Linux x64

Steps to Reproduce

Standalone script against the pinned vectordb version (same predicate strings as the source):

const lancedb = require("vectordb");
const fs = require("fs");
const DIR = fs.mkdtempSync("/tmp/lance-quote-");

async function main() {
  const db = await lancedb.connect(DIR);
  const makeArrowTable = lancedb.makeArrowTable;
  const rows = [
    { uuid: "u-1", cachekey: "ck-1", path: "src/normal.ts", vector: [1, 0], startLine: 1, endLine: 2, contents: "a" },
    { uuid: "u-2", cachekey: "ck-2", path: "src/don't.ts", vector: [0, 1], startLine: 1, endLine: 2, contents: "b" },
  ];
  const table = await db.createTable("chunks", await makeArrowTable(rows));

  // The exact predicate construction from LanceDbIndex.ts
  await table.delete(`cachekey = 'ck-2' AND path = 'src/normal.ts'`); // ok
  await table.delete(`cachekey = 'ck-2' AND path = 'src/don't.ts'`);  // throws
}
main().catch((e) => { console.error(String(e.message).split("\n")[0]); process.exit(1); });

Observed output:

apostrophe-path delete:      FAILED -> lance error: LanceError(IO): Unterminated string literal at Line: 1, Column 65 …
apostrophe-directory (LIKE): FAILED -> lance error: LanceError(IO): Unterminated string literal at Line: 1, Column 58 …
escaped variant:             ok

normal-path delete and the ''-escaped delete both succeed, isolating the interpolation as the cause.

Expected Behavior

Filesystem paths are arbitrary byte strings (modulo / and NUL); predicates built from them must be escaped (e.g. double the single quotes: 'don''t.ts' — verified to work against vectordb@0.4.20), or the filters should use a parameterized API. All interpolated sites in LanceDbIndex.ts (update() delete loop, _retrieveForTag path LIKE) need it; other artifacts (FullTextSearchCodebaseIndex, sqlite) already use bound parameters and are unaffected.

Happy to provide the full standalone script (including the retrieval-side case) or a patch.