Cloud: CloudFileSystem — synchronous IFileSystem over Firestore + Storage

Open
#792 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
firebase, typescript, wasm

Research direction

Start with packages/shell-core/src/types.ts and pathUtils.ts, then compare packages/shell-core/src/FileSystemAccessAPIFileSystem.ts with lua-learning-website/src/hooks/virtualFileSystemFactory.ts. Read src/hooks/virtualFileSystemStorage.ts, cloudFileSystemStorage.ts, and cloudFileSystemFactory.ts alongside the hydration, batching, retry, and emulator-testing requirements; done means all listed acceptance criteria pass without changing IFileSystem.

Written by the indexing model from the issue text.

Description

enhancement

Summary

Implement CloudFileSystem — a synchronous IFileSystem backed by Firestore (text) and
Cloud Storage (binary). This is the technical core of the milestone.

The constraint

IFileSystem (packages/shell-core/src/types.ts) is 100% synchronous. Every method returns a
value, not a Promise, because Lua's io.open, require, and every shell command call it
synchronously from inside the WASM engine. IFileSystem must not change.

The codebase already solves this exact problem twice, and the cloud backend is a third instance of
the same shape:

async hydrate -> full in-memory cache -> synchronous facade over the cache -> write-behind queue -> async flush()

See packages/shell-core/src/FileSystemAccessAPIFileSystem.ts and
lua-learning-website/src/hooks/virtualFileSystemFactory.ts. The async lifecycle rides on the
optional initialize() / flush() / refresh() methods that call sites already duck-type, so
no shared interface needs widening.

Batched, streaming hydration

Eagerly loading a whole 25 MB drive before the UI responds is not acceptable. Instead:

  1. One getDoc on users/{uid}/drive/manifest returns every path plus its size. The full
    tree renders immediately, with a real progress denominator
    ("Loading 47 of 210 files..."), reusing the existing pendingWorkspaces: Set<string>
    in useWorkspaceManager and the existing TreeNode.isLoading flag.
  2. Stream content in pages of ~200 documents (orderBy(__name__) + limit + startAfter),
    marking files loaded as pages arrive.
  3. Mount once text hydration completes. Binary blobs stay lazy and prefetch in the background.
  4. On repeat visits persistentLocalCache serves all of this from IndexedDB at zero read
    cost
    unless documents actually changed.

Storage split

  • Text <= 256 KB -> inline in users/{uid}/content/{encPath}.
  • Binary, or text > 256 KB -> Cloud Storage blob, with the manifest entry holding a pointer.
    The spill path means no file size ever hard-fails against Firestore's 1 MiB document limit.

Write path

Flush as a Firestore writeBatch containing both the content documents and the manifest
update
, so usedBytes / fileCount can never drift from what actually exists. Batches are
atomic and capped at 500 operations — chunk beyond that.

Acceptance criteria

  • IFileSystem in packages/shell-core is unchanged
  • src/hooks/cloudFileSystemStorage.ts mirrors the exported shape of virtualFileSystemStorage.ts
  • src/hooks/cloudFileSystemFactory.ts exports CloudFileSystemExtended with initialize(), flush(), isInitialized
  • Manifest-first hydration renders the tree from a single read
  • Content streams in pages with observable progress
  • Text > 256 KB and all binary spill to Cloud Storage transparently
  • Writes flush as an atomic writeBatch including the manifest; >500 ops chunk correctly
  • Write-behind queue retries and keeps a dirty set — failed operations are never dropped
  • readFile / require / io.open resolve synchronously against the cache once hydrated
  • Path handling reuses packages/shell-core/src/pathUtils.ts
  • encodeURIComponent path encoding guards ., .., and the 1500-byte doc-ID limit
  • Tested against the Firestore + Storage emulators, including a ~500-file workspace
  • Scoped mutation coverage >= 80%

Implementation notes and pre-existing bugs to avoid repeating

  • virtualFileSystemFactory.ts:181-183 drops failed operations from the queue after logging.
    For IndexedDB that is a rare edge case; over a network it is routine and would be silent data
    loss. The cloud queue needs real retry with backoff plus a dirty set that survives failure.
  • FileSystemAccessAPIFileSystem hydrates binary eagerly via await file.arrayBuffer() per file.
    Do not copy that for cloud — binary is lazy here.
  • FileOperationsHandler.fileClose (packages/lua-runtime/src/FileOperationsHandler.ts) is the
    one async point in the Lua file API (close returns a Promise so flush() can be awaited).
    That is the natural cloud commit point for Lua-side writes.
  • StoredFile in virtualFileSystemStorage.ts already carries createdAt/updatedAt but does
    not surface them through IFileSystem. The manifest carries hash + rev for the same reason —
    conflict detection in #796 needs it.

Depends on

  • #789

Branching

This issue is part of the Cloud Accounts & Storage
milestone, which uses feat/cloud as its integration branch.

Branch from feat/cloud, and open the PR against feat/cloud. Never against main.

Create the branch (the --base flag also sets the default PR target for this branch):

gh issue develop <number> --base feat/cloud --checkout

Open the PR:

python3 scripts/issue-review.py <number> --base feat/cloud \
  --summary "<summary>" --test-plan "<test-plan>"

feat/cloud merges to main in a single PR once the milestone is complete.

Why both flags are required: gh issue develop without --base branches from the repository
default branch (main), and scripts/issue-review.py falls back to main when --base is
omitted — both for the gh pr create target and for the changed-file list in the PR body.

Dominant language
TypeScript
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from jcollard/LuaInTheWeb

All issues in jcollard/LuaInTheWeb

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.