A new, modular, vibe code friendly architecture for tlparse
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 20/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- javascript, react, rust
- Domain
- backend, developer-experience, documentation, frontend, security, testing
Research direction
Start with linked issue #78 and the existing tlparse trace-processing and frontend architecture; this issue names no files or tests. Review the raw TORCH_TRACE and postprocessed JSON flow, the Manifold proxy constraints, React hosting questions, and ScubaDuck testing approach. Done requires an agreed modular viewer boundary and documented hosting, testing, security, and JSON-format decisions.
Written by the indexing model from the issue text.
Description
Redux of https://github.com/pytorch/tlparse/issues/78
I've been thinking about how to make tlparse more vibe coding friendly. Specifically, I want to make it easier for people to add custom, one-off frontends to view the data in tlparse, without having to do a full deploy process. My vision is to have a GitHub-like ecosystem of tlparse viewers which are maintained by individual people for their own use cases. People should be able to easily fork and customize viewers for their needs, and we can have a set of popular viewers that we can recommend to people by default (and maybe, eventually, a set of "official" viewers maintained by a core team).
To do this, we must define what the abstraction boundary that vibe code apps will interact with. A logical choice that requires no work is for vibe code viewers to directly interact with the raw TORCH_TRACE files that are generated. However, I think this is unwise: for example, a raw trace from a minified version of one of our production models is 150 MB! While this is not unmanageable, at this size it will take nontrivial time to load the trace in thef irst place.
The next most obvious fix is to separate out artifacts from the log stream, so that you have to do a separate HTTP call to fetch actual intermediate build products like Inductor code dumps. On the log I'm looking at, this reduces the trace size to 22M, with 40k rows. Discussion from https://x.com/pavpanchekha/status/1756070960837660949 indicates that at this size, rendering and not raw processing is the bottleneck. Frontends should either further filter this down, or make sure to implement some sort of virtual rendering functionality (I'm not clear if this can be conveniently done with vibe coding.)
Beyond this, my only other ideas is that you can do some further preprocessing in Rust itself, but it's not clear to me how we'd negotiate that.
For the actual architecture for vibe coded frontends, there are a still constraints to mindful of:
- The most convenient place for the postprocessed JSON files to live is on Manifold. This cannot be directly accessed by a client side JS application as there are no CORS headers, so you need to host a proxy that can access it. The proxy serve can/should be shared by everyone.
- Plain HTML/JS is relatively easy to scaffold, but if people prefer writing/vibe coding things in React we need to make some decisions about how the actual hosting works (there's an interaction with the proxy as well). A template for React is a lot more pressing.
- Should have web testing, similar to ScubaDuck. Public traces that we can test against are a must, ideally hosted in some centralized location so they can be reused. (Plain separate Git repo is probably OK, LFS probably not necessary?) Maybe also want a private store as well for internal only testing.
- Ideally, vibe coding can be directly on laptop or in public cloud, so external SOTA tools (claude code, codex can be used).
- Need some sort of "GitHub" fork/clone functionality, as well as app hosting. Actually, it wouldn't be crazy to do the hosting entirely with GitHub pages, with you running the proxy locally as needed. The proxy would need some specific CORS settings for safety. Ideally some sort of Metamate-based whitelist? If the ecosystem goes public, you need to audit deployments because they can exfiltrate sensitive data. Meta employees would be generally safe. (Also, you want to be able to run things locally!)
- We will need a comprehensive documentation about the JSON format which can be fed in as context to the LLM.
- Dominant language
- Rust
- Stars
- 94
- Forks
- 28
- PR merge metrics
- No merged PRs in 30d
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from meta-pytorch/tlparse
-
Difficulty 3/5 1-2 days Newbie friendliness 45/100
meta-pytorch/tlparse#162 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
meta-pytorch/tlparse#161 · 5 comments · 1 reaction ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 45/100
meta-pytorch/tlparse#153 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 38/100
meta-pytorch/tlparse#142 · 1 reaction ·
-
Difficulty 3/5 1-2 days Newbie friendliness 42/100
meta-pytorch/tlparse#141 · 1 comment ·
All issues in meta-pytorch/tlparse
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
linebender/parley#849 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
boxlite-ai/boxlite#1729 ·
Maintainers usually reply within 1 day
-
external
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
langchain-ai/docs#6255 ·
Maintainers usually reply within 1 day
-
P2 testing
Difficulty 1/5 Under an hour Newbie friendliness 90/100
Maintainers usually reply within 1 day