performance benchmarking / optimizing Greenwood

Type of Change

• New Feature Request
• Documentation / Website
• Improvement / Suggestion
• Bug
• Other (please clarify below)

Summary

Can't say I've focused to heavily on build time performances but since Zach Leatherman of 11ty put together a bench mark blog post and repo of some of the top SSGs, and so I figured I would test Greenwood just at least get a sense of where it sat amongst the rest.

I made a fork and decided to add Greenwood. 😬 🤞

I tried it on two machines and both had varying struggles:

• MBP 2016 (Intel) - Interestingly enough, was mostly able to sustain builds through the 1000 -> 2000 -> 4000 range, albeit it took a while 🐢 😁 . But still had some issues here and there. I was able to get a 4000 page run on Node 14, fwiw
• MBA 2020 (M1) - Had some OOO memory issues sometimes when doing 50 / 100 pages intermittently 😞

#
# Fatal error in , line 0
# Check failed: result.second.
#
#
#
#FailureMessage Object: 0x50651b1e8
 1: 0x1041a5874 node::NodePlatform::GetStackTracePrinter()::$_3::__invoke() [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
 2: 0x104f46d10 V8_Fatal(char const*, ...) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
 3: 0x10452b50c v8::internal::GlobalBackingStoreRegistry::Register(std::__1::shared_ptr<v8::internal::BackingStore>) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
 4: 0x1042a7c44 v8::ArrayBuffer::GetBackingStore() [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
 5: 0x1041241f0 node::Buffer::Data(v8::Local<v8::Value>) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
 6: 0x10414ed20 node::fs::Read(v8::FunctionCallbackInfo<v8::Value> const&) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
 7: 0x104301d30 v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
 8: 0x104301828 v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
 9: 0x1043010b4 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
10: 0x104a41b8c Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
11: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
12: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
13: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
14: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
15: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
16: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
17: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
18: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
19: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
20: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
21: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
22: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
23: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
24: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
25: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
26: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
27: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
28: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
29: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
30: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
31: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
32: 0x1049d34cc Builtins_JSEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
33: 0x1049d3164 Builtins_JSEntry [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
34: 0x1043a4d1c v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
35: 0x1043a43b0 v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
36: 0x1042b1840 v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
37: 0x104110908 node::loader::ModuleWrap::SyntheticModuleEvaluationStepsCallback(v8::Local<v8::Context>, v8::Local<v8::Module>) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
38: 0x10466afb8 v8::internal::SyntheticModule::Evaluate(v8::internal::Isolate*, v8::internal::Handle<v8::internal::SyntheticModule>) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
39: 0x10465595c v8::internal::SourceTextModule::InnerModuleEvaluation(v8::internal::Isolate*, v8::internal::Handle<v8::internal::SourceTextModule>, v8::internal::ZoneForwardList<v8::internal::Handle<v8::internal::SourceTextModule> >*, unsigned int*) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
40: 0x10465599c v8::internal::SourceTextModule::InnerModuleEvaluation(v8::internal::Isolate*, v8::internal::Handle<v8::internal::SourceTextModule>, v8::internal::ZoneForwardList<v8::internal::Handle<v8::internal::SourceTextModule> >*, unsigned int*) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
41: 0x10465564c v8::internal::SourceTextModule::Evaluate(v8::internal::Isolate*, v8::internal::Handle<v8::internal::SourceTextModule>) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
42: 0x10465551c v8::internal::SourceTextModule::EvaluateMaybeAsync(v8::internal::Isolate*, v8::internal::Handle<v8::internal::SourceTextModule>) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
43: 0x10429e9a8 v8::Module::Evaluate(v8::Local<v8::Context>) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
44: 0x104111edc node::loader::ModuleWrap::Evaluate(v8::FunctionCallbackInfo<v8::Value> const&) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
45: 0x104301d30 v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
46: 0x104301828 v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
47: 0x1043010b4 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
48: 0x104a41b8c Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
49: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
50: 0x104a04cd4 Builtins_AsyncFunctionAwaitResolveClosure [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
51: 0x104a89b78 Builtins_PromiseFulfillReactionJob [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
52: 0x1049f70b4 Builtins_RunMicrotasks [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
53: 0x1049d33a4 Builtins_JSRunMicrotasksEntry [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
54: 0x1043a4ce4 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
55: 0x1043a5118 v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
56: 0x1043a5204 v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*, v8::internal::MaybeHandle<v8::internal::Object>*) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
57: 0x1043c7e40 v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
58: 0x1043c86d4 v8::internal::MicrotaskQueue::PerformCheckpoint(v8::Isolate*) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
59: 0x104092db4 node::InternalCallbackScope::Close() [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
60: 0x1040927c4 node::InternalCallbackScope::~InternalCallbackScope() [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
61: 0x1041491d0 node::fs::FileHandle::CloseReq::Resolve() [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
62: 0x10415bf00 node::fs::FileHandle::ClosePromise()::$_0::__invoke(uv_fs_s*) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
63: 0x1049b3a5c uv__work_done [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
64: 0x1049b71f8 uv__async_io [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
65: 0x1049c8f70 uv__io_poll [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
66: 0x1049b7688 uv_run [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
67: 0x104093ccc node::SpinEventLoop(node::Environment*) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
68: 0x1041d8598 node::worker::Worker::Run() [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
69: 0x1041db7c0 node::worker::Worker::StartThread(v8::FunctionCallbackInfo<v8::Value> const&)::$_3::__invoke(void*) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
70: 0x1ba24226c _pthread_start [/usr/lib/system/libsystem_pthread.dylib]
71: 0x1ba23d08c thread_start [/usr/lib/system/libsystem_pthread.dylib]
./bench-greenwood.sh: line 17: 97189 Trace/BPT trap: 5       npx @greenwood/cli build

or

Here are specs, using the same approach Zach took. He didn't mention Node version, so for mine I used 16.14.0. The metric to track is real.

Each run was repeated 3 times and the lowest/fastest time was selected. This result set was generated on a MacBook Air (M1, 2020), macOS Monterey 12.5 12.3, 16 8 GB memory.

Results

MBP ⚠️

250

real	0m14.058s
user	0m37.615s
sys	0m6.054s

500

real	0m34.519s
user	1m20.436s
sys	0m14.706s

1000

real	1m6.309s
user	2m36.338s
sys	0m29.113s

2000

real	3m4.827s
user	5m56.250s
sys	1m16.211s

4000

🚫

MBA 🚨

25

real	0m1.303s
user	0m3.432s
sys	0m0.437s

50

real	0m1.762s
user	0m5.776s
sys	0m0.796s

100

real	0m2.980s
user	0m10.943s
sys	0m1.712s

250

real	0m5.540s
user	0m24.713s
sys	0m3.614s

500

🚫

1000

🚫

2000

🚫

4000

🚫

Details

So yeah, definitely some work we could / should do here, at least to chip away it for 1.0. Aside from the issues with the M1 chip one area of opportunity for sure seems to be the bundling phase as each benchmark run always seems to hang a little bit on this part of the process

success, done generating all pages...

Anyway, not sure what is realistic but would certainly like to be somewhat "competitive", as for v1.

Thoughts / Next Steps

1. Confirm I am not running Node via Rosetta (for M1) - https://stackoverflow.com/a/65449002/417806
2. Could be due to Worker threads, try with prerender: false - https://github.com/ProjectEvergreen/greenwood/pull/983
3. Bisect lifecycles (e.g. no bundle step) and see if that makes a difference. I know we do a lot of duplicate work in the bundle lifecycle which would be good to refactor out anyway. - #971
4. Stamp out unique page templates only once, re-use between page renders?
5. Throttle max pages to build at once? Or just have it be a Promise.all free for all? - https://github.com/ProjectEvergreen/greenwood/pull/983
6. Out of even more curiosity, I wonder would it would look like to benchmark the 0.25.0 version, to see how the results would have come out if using Puppeteer. 😅
7. Between prerender and bundle lifecycles, could we reduce "walks" over compilation.graph when iterating over pages. Maybe a Map / Set would be better here? - #971
8. Does multiplying / increasing number of workers in the poll hurt or help?
9. So... got cache busting to work for ESM, wonder if maybe we don't need to use Worker Threads anymore? (At least for production builds - https://github.com/ProjectEvergreen/greenwood/pull/1110)
10. After removing Worker threads for https://github.com/ProjectEvergreen/greenwood/pull/1110, wonder if we can do this as well for thread pooling? Or would we still want Workers in some capacity?
11. Per #1485 , I wonder if we need to run a benchmark with SSR pages to stress test the graph and make sure we don't need to be thread pooling in more places
    • additionally, is there any way we can bring down the overhead of spinning up plugins? (maybe pre-instantiate them during the config phase?)
    • are we missing areas where we should be thread pooling?
12. how to better handle async plugin processing in (development) server - https://github.com/ProjectEvergreen/greenwood/issues/1572
13. https://github.com/ProjectEvergreen/greenwood/issues/1575