good first issueperformance-improvement
仓库指标
- Star
- (15,558 star)
- PR 合并指标
- (平均合并 34天 14小时) (30 天内合并 120 个 PR)
描述
Expected Behavior or Use Case
Our testing infrastructure makes heavy use of the DistributedQueryRunner class such as in the IcebergQueryRunner and HiveQueryRunner. The startup time of the query runner contributes a respectable portion of the testing time. I think there are some optimizations that can be made.
- I noticed that everything is started serially. We could try to improve the overall startup time by parallelizing the launching of the workers and coordinator once the discovery server starts.
- There are other bits during startup which contribute a significant amount of time that could ideally be reduced or amortized.
BuiltInFunctionAndTypeManagertakes >700ms to initialize for every worker. Each worker initializes all types and compiles all built in functions every startup. It would be nice to turn this into a singleton so that each worker doesn't have to initialize this.
- It's common to initalize TPC-H/DS tables. Generally this is done serially. It can potentially be sped up by copying the tables concurrently.
- I noticed during profiling that one piece that takes a long time to initialize is the static
io.airlift.tpch.TextPool.getDefaultTestPool(). This generates a bunch of random text which is cached when generating tpch tables for the connector. The initialization of this component could be done proactively during server startup to improve TPC-H table initialization - Lastly (this is MacOS specific), make sure that
/etc/hostscontains a hostname other thanlocalhostfor the127.0.0.1and::1entries. This is due to some initialization logic in netty which hangs while looking up DNS names for the machine.
127.0.0.1 localhost <YOUR HOSTNAME>
255.255.255.255 broadcasthost
::1 localhost <YOUR HOSTNAME>
With changes 1, 3, 4, and 5 I was able to get my local environment to launch test clusters in 7-8s. Previously it was 15s. About a 50% improvement. 4s was from the DNS fix, ~2s from parallelizing worker launch. ~1s for parallelizing table copies, ~1s for async init of TPC-H text pool
Presto Component, Service, or Connector
DistributedQueryRunner
Possible Implementation
See #22212
Example Screenshots (if appropriate):
Context
Bothered at waiting so long for some query runners to start up