grafana/k6

Optimize tests run speed (especially the functional tests)

Open

#4412 opened on Oct 27, 2022

View on GitHub
 (4 comments) (0 reactions) (0 assignees)Go (30,564 stars) (1,537 forks)batch import
area: browsergood first issuehacktoberfestteststriage

Description

This can be ongoing work. One idea is:

  • Having a pool of browsers
  • Sharing the pool among the tests (they don't run in parallel 10-20 tests at a time (depending on the number of cores on your machine—on the other side, it won't matter for CI. So the solution below is needed too.)
  • Optimizing tests further

Advantages:

  • Less local wait times
  • Less CI wait times
  • Can teach us how to work with a pool of browsers (pre-AWS browser farms?)

Here is the current Go tests run speeds:

Test Sec
TestPageScreenshotFullpage 2.53
TestPageWaitForFunction/ok_func_poll_mutation 1.76
TestPageWaitForFunction/ok_expr_poll_interval 1.74
TestLifecycleNetworkIdle/doesn't_unblock... 1.69
TestWaitForFrameNavigationWithinDocument/anchor 1.68
TestWaitForFrameNavigationWithinDocument/history 1.64
TestLocator/Hover 1.57
TestKeyboardPress 1.56
TestLocator/DblClick 1.48
TestPageWaitForFunction/ok_func_raf_default 1.46
TestLaunchOptionsSlowMo/Frame/uncheck 1.32
TestElementHandleClickConcealedLink 1.31
TestElementHandleNonClickable 1.29
TestLaunchOptionsSlowMo/Frame/hover 1.26
TestLaunchOptionsSlowMo/Frame/check 1.25
TestLifecycleNetworkIdle/doesn't_timeout_waiting_for_networkIdle 1.24
TestLocator/Click 1.22
TestWaitForFrameNavigation 1.17
TestPageWaitForFunction/err_expr_raf_timeout 1.13
TestLocator/Tap 1.12
TestLaunchOptionsSlowMo/Page/check 1.12
TestLaunchOptionsSlowMo/Frame/click 1.12
TestLocatorElementState/hidden 1.11
TestLaunchOptionsSlowMo/Page/uncheck 1.11
TestLocator/strict/InputValue 1.1
TestLocator/strict/DispatchEvent 1.1
TestLocator/DispatchEvent 1.1
TestPageContent 1.08
TestLocatorElementState/timeout/IsDisabled 1.08
TestLocator/timeout/WaitFor 1.08
TestLocator/InnerHTML 1.08
TestLocator/timeout/Hover 1.06
TestLocator/Focus 1.06
TestLaunchOptionsSlowMo/Frame/dblClick 1.06
TestPageWaitForLoadState/err_wrong_event 1.05
TestLocator/timeout/Type 1.05
TestLocator/strict/Press 1.05
TestLocator/strict/InnerText 1.05
TestLocator/strict/GetAttribute 1.05
TestLocator/Press 1.05
TestLocator/InnerText 1.05
TestLocator/strict/TextContent 1.04
TestLocatorElementState/readOnly 1.03
TestLocator/strict/Hover 1.03
TestLocatorElementState/timeout/IsChecked 1.02
TestLocatorElementState/disabled 1.02
TestLocator/strict/WaitFor 1.01
TestLocator/strict/Focus 1.01
TestLocator/GetAttribute 1.01
TestPageEvaluate/ok/func_arg 0.99
TestLocator/timeout/DispatchEvent 0.99
TestLocator/strict/Tap 0.99
TestLocator/timeout/SelectOption 0.97
TestLocator/timeout/Fill 0.97
TestLocator/timeout/Check 0.97
TestLocator/strict/Uncheck 0.97
TestLocator/strict/Type 0.97
TestLocator/Fill 0.97
TestLaunchOptionsSlowMo/Page/hover 0.97
TestURLSkipRequest 0.96
TestLocator/strict/Dblclick 0.96
TestLaunchOptionsSlowMo/Frame/press 0.96
TestLocator/strict/SelectOption 0.95
TestLaunchOptionsSlowMo/Frame/setContent 0.95
TestLocator/timeout/Uncheck 0.94
TestLocator/strict/InnerHTML 0.94
TestLocator/strict/Fill 0.94
TestPageInnerText/err_wrong_selector 0.93
TestLaunchOptionsSlowMo/Frame/evaluate 0.93
TestLocator/SelectOption 0.92
TestLaunchOptionsSlowMo/Frame/evaluateHandle 0.92
TestLocator/timeout/Focus 0.91
TestLocator/strict/Check 0.91
TestLaunchOptionsSlowMo/Page/type 0.91
TestLocator/Type 0.9
TestLaunchOptionsSlowMo/Page/dblClick 0.89
TestLaunchOptionsSlowMo/Page/click 0.89
TestLaunchOptionsSlowMo/Page/setViewportSize 0.88
TestLaunchOptionsSlowMo/Page/press 0.88
TestPageInnerText/err_empty_selector 0.87
TestLaunchOptionsSlowMo/Frame/selectOption 0.87
TestLaunchOptionsSlowMo/Frame/focus 0.87
TestLocator/timeout/Tap 0.86
TestLocator/timeout/GetAttribute 0.86
TestLocator/strict/Click 0.86
TestLaunchOptionsSlowMo/Frame/goto 0.86
TestLocator/timeout/TextContent 0.85
TestLocator/WaitFor 0.85
TestLocatorElementState/strict/IsVisible 0.84
TestLocator/timeout/Dblclick 0.84
TestLaunchOptionsSlowMo/Page/fill 0.83
TestLocatorElementState/visible 0.82
TestLocatorElementState/strict/IsEnabled 0.82
TestLocatorElementState/strict/IsDisabled 0.82
TestLocator/Check 0.81
TestLaunchOptionsSlowMo/Page/focus 0.81
TestPageInnerText/ok 0.8
TestLaunchOptionsSlowMo/Page/selectOption 0.8
TestLaunchOptionsSlowMo/Page/evaluate 0.8
TestLaunchOptionsSlowMo/Frame/fill 0.8
TestLocator/timeout/Press 0.79
TestLaunchOptionsSlowMo/Page/setContent 0.79
TestLaunchOptionsSlowMo/Frame/dispatchEvent 0.79
TestElementHandleClickWithNodeRemoved 0.79
TestPageWaitForFunction/ok_func_raf_default_arg 0.78
TestLocatorElementState/timeout/IsEnabled 0.78
TestLocatorElementState/strict/IsChecked 0.78
TestLocator/timeout/InputValue 0.78
TestLocator/timeout/InnerHTML 0.78
TestLaunchOptionsSlowMo/Page/dispatchEvent 0.78
TestLaunchOptionsSlowMo/Frame/type 0.78
TestLaunchOptionsSlowMo/Page/evaluateHandle 0.77
TestLaunchOptionsSlowMo/Page/emulateMedia 0.77
TestLocator/timeout/Click 0.76
TestPageWaitForFunction/ok_func_raf_default_args 0.75
TestLocatorElementState/timeout/IsVisible 0.75
TestPageInnerHTML/err_empty_selector 0.74
TestElementHandleClick 0.74
TestLaunchOptionsSlowMo/Page/goto 0.73
TestPageInnerHTML/err_wrong_selector 0.72
TestPageTextContent/err_wrong_selector 0.7
TestLocatorElementState/enabled 0.67
TestLocator/InputValue 0.67
TestLocatorElementState/strict/IsEditable 0.66
TestPageInnerHTML/ok 0.65
TestLocatorElementState/timeout/IsHidden 0.65
TestLocatorElementState/strict/IsHidden 0.65
TestElementHandleScreenshot 0.64
TestPageTextContent/ok 0.63
TestPageEvaluate/err/syntax 0.63
TestLocatorElementState/timeout/IsEditable 0.63
TestLaunchOptionsSlowMo/Page/reload 0.62
TestElementHandleWaitForSelector 0.6
TestLocator/timeout/InnerText 0.57
TestLocator/TextContent 0.57
TestPageTextContent/err_empty_selector 0.56
TestPageWaitForFunction/err_wrong_polling 0.55
TestPageEvaluate/err/undef 0.54
TestPageEvaluate/err/promise 0.54
TestPageInputSpecialCharacters 0.49
TestElementHandleBoundingBoxSVG 0.48
TestElementHandleClickWithDetachedNode 0.47
TestBrowserOn/ok_promise_resolved 0.47
TestPageIsChecked 0.46
TestPageEmulateMedia 0.46
TestElementHandleIsChecked 0.46
TestPageFill 0.45
TestFramePress 0.45
TestPagePress 0.44
TestBrowserOn/ok_promise_rejected 0.44
TestPageInputValue 0.43
TestElementHandleQueryAll 0.43
TestElementHandlePress 0.43
TestLocatorPress 0.42
TestElementHandleBoundingBoxInvisibleElement 0.42
TestPageTitle 0.41
TestElementHandleGetAttribute 0.41
TestElementHandleInputValue 0.4
TestBlockIPs 0.4
TestBlockHostnames 0.39
TestBrowserLogIterationID 0.38
TestPageGotoWaitUntilDOMContentLoaded 0.37
TestTestBrowserAwaitWithTimeoutShortCircuit 0.36
TestBasicAuth 0.36
TestPageGotoWaitUntilLoad 0.35
TestBrowserContextOptionsDefaultViewport 0.35
TestJSHandleGetProperties 0.34
TestBrowserOn/err_wrong_event 0.34
TestBrowserNewPage 0.33
TestPageSetExtraHTTPHeaders 0.31
TestPageGoto 0.31
TestBrowserContextOptionsExtraHTTPHeaders 0.31
TestTmpDirCleanup 0.3
TestPageGotoDataURI 0.29
TestBrowserContextOptionsSetViewport 0.29
TestPageWaitForNavigationShouldNotPanic 0.28
TestBrowserNewPageInContext/timeout 0.2
TestKeyboardPress/all_keys 0.19
TestKeyboardPress/selection 0.18
TestKeyboardPress/combo 0.18
TestKeyboardPress/backspace 0.18
TestKeyboardPress/type_does_not_split_on_+ 0.17
TestKeyboardPress/newline 0.17
TestBrowserUserAgent 0.17
TestKeyboardPress/type_not_affected_by_shift 0.16
TestKeyboardPress/capitalization 0.16
TestBasicAuth/valid 0.16
TestBrowserVersion 0.13
TestFrameNilDocument 0.1
TestPageFill/happy/text 0.07
TestBrowserCrashErr 0.07
TestElementHandleQueryAll/element_handle 0.06
TestOnRequestPausedBlockedIPs 0.05
TestBasicAuth/invalid 0.05
TestLocator 0.04
TestOnRequestPausedBlockedHostnames 0.03
TestNetworkManagerEmitRequestResponseMetricsTimingSkew/ts_past 0.03
TestOnRequestPausedBlockedIPs/ok_continue_empty 0.02
TestOnRequestPausedBlockedHostnames/ok_fail_simple 0.02
TestLaunchOptionsParse/slowMo 0.02
TestLaunchOptionsParse/logCategoryFilter 0.02
TestLaunchOptionsParse/env_err 0.02
TestLaunchOptionsParse/args 0.02
TestValueFromRemoteObject 0.01
TestRequest 0.01
TestOnRequestPausedBlockedIPs/ok_fail_resolved_ip 0.01
TestNetworkManagerEmitRequestResponseMetricsTimingSkew/ok2 0.01
TestNetworkManagerEmitRequestResponseMetricsTimingSkew/ok 0.01
TestLaunchOptionsParse/timeout_err 0.01
TestLaunchOptionsParse/proxy_err 0.01
TestLaunchOptionsParse/headless_err 0.01
TestLaunchOptionsParse/devtools 0.01
TestFrameWaitForNavigationOptionsParse/err/invalid_waitUntil 0.01
TestConvertArgument 0.01
TestBaseEventEmitter/order_of_emitted_events_kept 0.01
TestBaseEventEmitter/order_of_emitted_different_event_types_kept 0.01
TestBaseEventEmitter/handler_can_emit_without_deadlocking 0.01
TestWaitForFrameNavigationWithinDocument 0
TestViewportCalculateInset/windows 0
TestViewportCalculateInset/linux 0
TestViewportCalculateInset/headless 0
TestViewportCalculateInset/headful 0
TestViewportCalculateInset/darwin 0
TestViewportCalculateInset 0
TestValueFromRemoteObject/unserializable_value_error 0
TestValueFromRemoteObject/remote_object_with_ID 0
TestValueFromRemoteObject/primitive_types 0
TestValueFromRemoteObject/float64_unserializable_values 0
TestValueFromRemoteObject/bigint_parsing_error 0
TestTimeoutSettings/TimeoutSettings.timeout/should_work_with_parent 0
TestTimeoutSettings/TimeoutSettings.timeout/should_work 0
TestTimeoutSettings/TimeoutSettings.timeout 0
TestTimeoutSettings/TimeoutSettings.setDefaultTimeout/should_work 0
TestTimeoutSettings/TimeoutSettings.setDefaultTimeout 0
TestTimeoutSettings/TimeoutSettings.setDefaultNavigationTimeout/should_work 0
TestTimeoutSettings/TimeoutSettings.setDefaultNavigationTimeout 0
TestTimeoutSettings/TimeoutSettings.navigationTimeout/should_work_with_parent 0
TestTimeoutSettings/TimeoutSettings.navigationTimeout/should_work 0
TestTimeoutSettings/TimeoutSettings.navigationTimeout 0
TestTimeoutSettings/TimeoutSettings.NewTimeoutSettings/should_work_with_parent 0
TestTimeoutSettings/TimeoutSettings.NewTimeoutSettings/should_work 0
TestTimeoutSettings/TimeoutSettings.NewTimeoutSettings 0
TestTimeoutSettings 0
TestSplit/string_split_with_separator 0
TestSplit/split_++to+and'' 0
TestSplit/split_+++to+and+ 0
TestSplit/empty_slice_on_string_without_separator 0
TestSplit/empty_slice_on_empty_string 0
TestSplit/do_not_split_on_single_+ 0
TestSplit 0
TestSessionCreateSession/send_and_recv_session_commands 0
TestSessionCreateSession 0
TestRequest/error_parse_url 0
TestRequest/Size() 0
TestRequest/HeadersArray() 0
TestRequest/Headers() 0
TestRequest/HeaderValue() 0
TestQueryAll/returns_elems 0
TestQueryAll/nil_handles_no_err 0
TestQueryAll/invalid_selector 0
TestQueryAll/invalid_js_handle 0
TestQueryAll/eval_call 0
TestQueryAll/disposes_main_handle 0
TestQueryAll/disposes_handles 0
TestQueryAll/cannot_evaluate 0
TestQueryAll 0
TestParseRemoteObject/nested 0
TestParseRemoteObject/most_types 0
TestParseRemoteObject/err_parsing_property 0
TestParseRemoteObject/err_overflow 0
TestParseRemoteObject 0
TestPageWaitForLoadState 0
TestPageWaitForFunction 0
TestPageTextContent 0
TestPageLocator 0
TestPageInnerText 0
TestPageInnerHTML 0
TestPageFill/sad/unfillable 0
TestPageFill/sad/number 0
TestPageFill/sad/date 0
TestPageFill/happy/number 0
TestPageFill/happy/date 0
TestPageEvaluate/err 0
TestPageEvaluate 0
TestOnRequestPausedBlockedIPs/ok_fail_simple 0
TestOnRequestPausedBlockedIPs/ok_continue_simple 0
TestOnRequestPausedBlockedIPs/ok_continue_resolved_ip 0
TestOnRequestPausedBlockedHostnames/ok_continue_simple 0
TestOnRequestPausedBlockedHostnames/ok_continue_ip 0
TestOnRequestPausedBlockedHostnames/ok_continue_empty 0
TestOnRequestPausedBlockedHostnames/err_url_continue 0
TestNetworkManagerEmitRequestResponseMetricsTimingSkew/wt_past 0
TestNetworkManagerEmitRequestResponseMetricsTimingSkew/wt_future 0
TestNetworkManagerEmitRequestResponseMetricsTimingSkew/ts_future 0
TestNetworkManagerEmitRequestResponseMetricsTimingSkew 0
TestLocatorElementState 0
TestLifecycleEventUnmarshalText/ok 0
TestLifecycleEventUnmarshalText/err/invalid_empty 0
TestLifecycleEventUnmarshalText/err/invalid 0
TestLifecycleEventUnmarshalText 0
TestLifecycleEventMarshalTextRound 0
TestLifecycleEventMarshalText/ok/nil 0
TestLifecycleEventMarshalText/err/invalid 0
TestLifecycleEventMarshalText 0
TestLaunchOptionsSlowMo/Page 0
TestLaunchOptionsSlowMo/Frame 0
TestLaunchOptionsSlowMo/ElementHandle 0
TestLaunchOptionsSlowMo 0
TestLaunchOptionsParse/timeout 0
TestLaunchOptionsParse/slowMo_err 0
TestLaunchOptionsParse/proxy 0
TestLaunchOptionsParse/logCategoryFilter_err 0
TestLaunchOptionsParse/ignoreDefaultArgs_err 0
TestLaunchOptionsParse/ignoreDefaultArgs 0
TestLaunchOptionsParse/headless 0
TestLaunchOptionsParse/executablePath_err 0
TestLaunchOptionsParse/executablePath 0
TestLaunchOptionsParse/env 0
TestLaunchOptionsParse/devtools_err 0
TestLaunchOptionsParse/defaults 0
TestLaunchOptionsParse/debug_err 0
TestLaunchOptionsParse/debug 0
TestLaunchOptionsParse/args_err 0
TestLaunchOptionsParse 0
TestKeyboardPress/panics_when_''_empty_key_passed_in 0
TestKeyboardPress 0
TestFrameWaitForNavigationOptionsParse/ok 0
TestFrameWaitForNavigationOptionsParse 0
TestFrameSetContentOptionsParse/ok 0
TestFrameSetContentOptionsParse/err/invalid_waitUntil 0
TestFrameSetContentOptionsParse 0
TestFrameManagerFrameAbortedNavigationShouldEmitANonNilPendingDocument 0
TestFrameGotoOptionsParse/ok 0
TestFrameGotoOptionsParse/err/invalid_waitUntil 0
TestFrameGotoOptionsParse 0
TestEventEmitterSpecificEvent/remove_event_handler 0
TestEventEmitterSpecificEvent/emit_event 0
TestEventEmitterSpecificEvent/add_event_handler 0
TestEventEmitterSpecificEvent 0
TestEventEmitterAllEvents/remove_catch-all_event_handler 0
TestEventEmitterAllEvents/emit_event 0
TestEventEmitterAllEvents/add_catch-all_event_handler 0
TestEventEmitterAllEvents 0
TestErrorFromDOMError 0
TestElementHandleQueryAll/page 0
TestElementHandleQueryAll/frame 0
TestConvertArgument/string 0
TestConvertArgument/int64_maxint 0
TestConvertArgument/int64 0
TestConvertArgument/float64_unserializable_values 0
TestConvertArgument/float64 0
TestConvertArgument/bool 0
TestConvertArgument/*BaseJSHandle_wrong_context 0
TestConvertArgument/*BaseJSHandle_is_disposed 0
TestConvertArgument/BaseJSHandle_as_ElementHandle 0
TestConvertArgument/*BaseJSHandle 0
TestContextWithDoneChan 0
TestConnectionSendRecv/send_command_with_empty_reply 0
TestConnectionSendRecv 0
TestConnectionCreateSession/create_session_for_target 0
TestConnectionCreateSession 0
TestConnectionClosureAbnormal/closure_abnormal 0
TestConnectionClosureAbnormal 0
TestConnection/connect 0
TestConnection 0
TestBrowserTypePrepareFlags/host-resolver-rules 0
TestBrowserTypePrepareFlags/host-resolver-rules#01 0
TestBrowserTypePrepareFlags/hide-scrollbars 0
TestBrowserTypePrepareFlags/hide-scrollbars#01 0
TestBrowserTypePrepareFlags/headless 0
TestBrowserTypePrepareFlags/enable-use-zoom-for-dsf 0
TestBrowserTypePrepareFlags/browser-args 0
TestBrowserTypePrepareFlags/browser-arg-trim-single-quote 0
TestBrowserTypePrepareFlags/browser-arg-trim-double-quote 0
TestBrowserTypePrepareFlags/browser-arg-flag 0
TestBrowserTypePrepareFlags/browser-arg 0
TestBrowserTypePrepareFlags/auto-open-devtools-for-tabs 0
TestBrowserTypePrepareFlags 0
TestBrowserOn 0
TestBrowserNewPageInContext/missing_browser_context 0
TestBrowserNewPageInContext/happy_path 0
TestBrowserNewPageInContext/error_in_create_target_action 0
TestBrowserNewPageInContext/context_done 0
TestBrowserNewPageInContext 0
TestBrowserContextOptionsPermissions 0
TestBrowserContextOptionsDefaultValues 0
TestBaseEventEmitter 0
TestBarrier 0

Contributor guide

Tech stack
go
Domain
testingperformance
Issue type
performance
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Go testing frameworkconcurrent test execution
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
35
Research direction
Start by reviewing the current test structure in the repository, focusing on how browsers are launched and managed. Investigate the possibility of creating a shared browser pool to run tests concurrently. Analyze the slowest tests (e.g., TestPageScreenshotFullpage) to identify bottlenecks. Consider implementing a simple browser pool and measuring the speed improvement.