Lakebase DX: a native-search helper, grants bootstrap, and graduating proven cookbook patterns

Open
  • Is something wrong?
#557 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
fastapi, postgresql, python, typescript
Domain
backend-api-design, databases, developer-experience

Research direction

Start by confirming the Lakebase Search GA/interface internally, then compare the proposed shape with AppKit’s existing ai-search helper. Review the cookbook’s migrations_runner.py and apps/lakebase-fastapi pattern, along with the proposed lakebase({ branch }) entry point, before splitting the request into scoped contributions. Done should be a maintainer-approved scope for search, grants, migrations, reusable scaffolding, and branch selection.

Written by the indexing model from the issue text.

Description

Field feedback from building an app on @databricks/appkit 0.66.1 + Lakebase. First, credit where it's due — these saved real work: the Lakebase plugin's auto OAuth refresh, asUser(req) OBO per-user pools, and exposeAsAgentTool (read-only SQL classifier). And ai-search is great for Databricks Vector Search.

The gaps below are all Lakebase-specific. Each is verified against 0.66.1; happy to contribute PRs.

1. A lakebaseSearch() helper (Lakebase-native BM25 + vector ANN)

ai-search targets Vector Search (managed indexes), not Lakebase's native lakebase_bm25 + lakebase_ann (halfvec) indexes. There's no helper for those, so devs write raw SQL and hit avoidable traps:

  • the bm25 regclass resolves against search_path, so from an app you must schema-qualify it (to_bm25query(..., 'app.providers_bm25')) or get relation "providers_bm25" does not exist;
  • plainto_tsquery, not to_tsquery (the latter crashes on multi-word input);
  • halfvec(1024) cosine + ai_query('databricks-gte-large-en', text) embeddings.

Proposed API, mirroring ai-search's shape but pointing at Lakebase indexes:

lakebaseSearch({
  indexes: { providers: { table: "app.providers", bm25Index: "app.providers_bm25",
    annIndex: "app.providers_ann", textColumn: "search_tsv",
    embeddingColumn: "embedding", mode: "hybrid" } }
})
// appkit.lakebaseSearch.query("providers", { text: "home health", numResults: 5 })

(Worth confirming Lakebase Search GA/interface internally before building.)

2. Service-principal grants bootstrap + a clear grant error

The app SP needs databricks_create_role(...) + explicit GRANTs; today that's manual copy-paste, and a missing grant surfaces as a generic permission denied. Proposal: a ensureServicePrincipalGrants(spClientId, { schema, tables }) helper (+ an optional apps init grants.sql scaffold), and map Postgres permission denied → "the app SP lacks GRANT X on Y".

3. Graduate proven Lakebase patterns from the cookbook into AppKit

databricks-solutions/lakebase-cookbook already has battle-tested code for these — the ask is to bring them into the toolkit:

  • Migrations: an appkit migrate wrapping the cookbook's checksum/drift migrations_runner.py, made multi-SP-ownership aware (CREATE INDEX IF NOT EXISTS still fails if another SP owns the table).
  • Synced-read + writable-ops scaffold: the apps/lakebase-fastapi pattern (read-only synced mirror + writable ops table + derive-state-by-join) as a reusable helper/template.
  • Dev-branch selection: a lakebase({ branch }) config so an app targets a dev branch vs production without hand-editing endpoint paths.
Dominant language
TypeScript
Stars
119
Forks
36
Avg merge
4d 16h
Merged PRs (30d)
37

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from databricks/appkit

All issues in databricks/appkit

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.