Cloud: CloudFileSystem — synchronous IFileSystem over Firestore + Storage
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- firebase, typescript, wasm
Research direction
Start with packages/shell-core/src/types.ts and pathUtils.ts, then compare packages/shell-core/src/FileSystemAccessAPIFileSystem.ts with lua-learning-website/src/hooks/virtualFileSystemFactory.ts. Read src/hooks/virtualFileSystemStorage.ts, cloudFileSystemStorage.ts, and cloudFileSystemFactory.ts alongside the hydration, batching, retry, and emulator-testing requirements; done means all listed acceptance criteria pass without changing IFileSystem.
Written by the indexing model from the issue text.
Description
Summary
Implement CloudFileSystem — a synchronous IFileSystem backed by Firestore (text) and
Cloud Storage (binary). This is the technical core of the milestone.
The constraint
IFileSystem (packages/shell-core/src/types.ts) is 100% synchronous. Every method returns a
value, not a Promise, because Lua's io.open, require, and every shell command call it
synchronously from inside the WASM engine. IFileSystem must not change.
The codebase already solves this exact problem twice, and the cloud backend is a third instance of
the same shape:
async hydrate -> full in-memory cache -> synchronous facade over the cache -> write-behind queue ->
async flush()
See packages/shell-core/src/FileSystemAccessAPIFileSystem.ts and
lua-learning-website/src/hooks/virtualFileSystemFactory.ts. The async lifecycle rides on the
optional initialize() / flush() / refresh() methods that call sites already duck-type, so
no shared interface needs widening.
Batched, streaming hydration
Eagerly loading a whole 25 MB drive before the UI responds is not acceptable. Instead:
- One
getDoconusers/{uid}/drive/manifestreturns every path plus its size. The full
tree renders immediately, with a real progress denominator
("Loading 47 of 210 files..."), reusing the existingpendingWorkspaces: Set<string>
inuseWorkspaceManagerand the existingTreeNode.isLoadingflag. - Stream content in pages of ~200 documents (
orderBy(__name__)+limit+startAfter),
marking files loaded as pages arrive. - Mount once text hydration completes. Binary blobs stay lazy and prefetch in the background.
- On repeat visits
persistentLocalCacheserves all of this from IndexedDB at zero read
cost unless documents actually changed.
Storage split
- Text <= 256 KB -> inline in
users/{uid}/content/{encPath}. - Binary, or text > 256 KB -> Cloud Storage blob, with the manifest entry holding a pointer.
The spill path means no file size ever hard-fails against Firestore's 1 MiB document limit.
Write path
Flush as a Firestore writeBatch containing both the content documents and the manifest
update, so usedBytes / fileCount can never drift from what actually exists. Batches are
atomic and capped at 500 operations — chunk beyond that.
Acceptance criteria
-
IFileSysteminpackages/shell-coreis unchanged -
src/hooks/cloudFileSystemStorage.tsmirrors the exported shape ofvirtualFileSystemStorage.ts -
src/hooks/cloudFileSystemFactory.tsexportsCloudFileSystemExtendedwithinitialize(),flush(),isInitialized - Manifest-first hydration renders the tree from a single read
- Content streams in pages with observable progress
- Text > 256 KB and all binary spill to Cloud Storage transparently
- Writes flush as an atomic
writeBatchincluding the manifest; >500 ops chunk correctly - Write-behind queue retries and keeps a dirty set — failed operations are never dropped
-
readFile/require/io.openresolve synchronously against the cache once hydrated - Path handling reuses
packages/shell-core/src/pathUtils.ts -
encodeURIComponentpath encoding guards.,.., and the 1500-byte doc-ID limit - Tested against the Firestore + Storage emulators, including a ~500-file workspace
- Scoped mutation coverage >= 80%
Implementation notes and pre-existing bugs to avoid repeating
virtualFileSystemFactory.ts:181-183drops failed operations from the queue after logging.
For IndexedDB that is a rare edge case; over a network it is routine and would be silent data
loss. The cloud queue needs real retry with backoff plus a dirty set that survives failure.FileSystemAccessAPIFileSystemhydrates binary eagerly viaawait file.arrayBuffer()per file.
Do not copy that for cloud — binary is lazy here.FileOperationsHandler.fileClose(packages/lua-runtime/src/FileOperationsHandler.ts) is the
one async point in the Lua file API (closereturns a Promise soflush()can be awaited).
That is the natural cloud commit point for Lua-side writes.StoredFileinvirtualFileSystemStorage.tsalready carriescreatedAt/updatedAtbut does
not surface them throughIFileSystem. The manifest carrieshash+revfor the same reason —
conflict detection in #796 needs it.
Depends on
- #789
Branching
This issue is part of the Cloud Accounts & Storage
milestone, which uses feat/cloud as its integration branch.
Branch from feat/cloud, and open the PR against feat/cloud. Never against main.
Create the branch (the --base flag also sets the default PR target for this branch):
gh issue develop <number> --base feat/cloud --checkout
Open the PR:
python3 scripts/issue-review.py <number> --base feat/cloud \
--summary "<summary>" --test-plan "<test-plan>"
feat/cloud merges to main in a single PR once the milestone is complete.
Why both flags are required: gh issue develop without --base branches from the repository
default branch (main), and scripts/issue-review.py falls back to main when --base is
omitted — both for the gh pr create target and for the changed-file list in the PR body.
- Dominant language
- TypeScript
- Stars
- 1
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from jcollard/LuaInTheWeb
-
tech-debt
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
jcollard/LuaInTheWeb#752 ·
-
tech-debt
Difficulty 1/5 Under an hour Newbie friendliness 68/100
jcollard/LuaInTheWeb#690 ·
-
tech-debt
Difficulty 1/5 Under an hour Newbie friendliness 75/100
jcollard/LuaInTheWeb#590 ·
-
tech-debt
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
jcollard/LuaInTheWeb#518 ·
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
jcollard/LuaInTheWeb#235 ·
All issues in jcollard/LuaInTheWeb
Similar issues
-
S: triage
Difficulty 1/5 Under an hour Newbie friendliness 85/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
fix(errors): EHOSTUNREACH from a happy-eyeballs connect is reported as a resolver error (STAMP-80) Open
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
snapshot-labs/stamp#666 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
GauravKarakoti/SecureFlow#1070 · 1 comment ·
-
comp/desktop P3 type/bug
Difficulty 1/5 Under an hour Newbie friendliness 92/100
NousResearch/hermes-agent#118866 ·