Stack trie inner_fmt is slow
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Refactor
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- rust
- Domain
- cli, performance
Research direction
Start by reproducing the large structured-log case with --plain-text and profiling the stack trie inner_fmt path using cargo flamegraph. Compare the formatting work added by commit 9cfe1d0fdd48fd1f820fe9f06a137207198724d1 on compilation metric pages, and consider whether Rust-side optimization or deferred JavaScript formatting provides a measurable improvement.
Written by the indexing model from the issue text.
Description
As tlparse outputs get larger, I start noticing that tlparse itself is slow for large (>100MiB) structured logs.
I did a quick profile using cargo flamegraph to see what takes the most time in tlparse (assuming --plain-text).
Here's the flamegraph:
https://github.com/pytorch/tlparse/commit/9cfe1d0fdd48fd1f820fe9f06a137207198724d1
adds stack trie mini stacks to every compilation metric page. Turns out, doing this is pretty slow overall.
I think internally, downloading/uploading is still the majority of the time spent in tlparse, so this isn't super urgent. But might worth considering if we can optimize this better (for example, instead of formatting on rust side, we could inject javascript to do the formatting when the user opens the page instead)
- Dominant language
- Rust
- Stars
- 94
- Forks
- 28
- PR merge metrics
- No merged PRs in 30d
Getting set up
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from meta-pytorch/tlparse
-
Difficulty 3/5 1-2 days Newbie friendliness 45/100
meta-pytorch/tlparse#162 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
meta-pytorch/tlparse#161 · 5 comments · 1 reaction ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 45/100
meta-pytorch/tlparse#153 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 38/100
meta-pytorch/tlparse#142 · 1 reaction ·
-
Difficulty 3/5 1-2 days Newbie friendliness 42/100
meta-pytorch/tlparse#141 · 1 comment ·
All issues in meta-pytorch/tlparse
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
trailofbits/dylint#2107 ·
Maintainers usually reply within 1 day
-
area:cli bug good first issue priority:medium
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
Maintainers usually reply within 1 day
-
arrays_zip with two same-named inputs fails with "ArrowArray struct has 2 children (expected 1)"Openbug requires-triage
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/datafusion-comet#6251 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
bug false-positive harper-core linting
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Automattic/harper#4471 ·
Maintainers usually reply within 1 day