Metriche repository
- Star
- (15.558 star)
- Metriche merge PR
- (Merge medio 34g 14h) (120 PR mergiate in 30 g)
Descrizione
Expected Behavior or Use Case
Our testing infrastructure makes heavy use of the DistributedQueryRunner class such as in the IcebergQueryRunner and HiveQueryRunner. The startup time of the query runner contributes a respectable portion of the testing time. I think there are some optimizations that can be made.
- I noticed that everything is started serially. We could try to improve the overall startup time by parallelizing the launching of the workers and coordinator once the discovery server starts.
- There are other bits during startup which contribute a significant amount of time that could ideally be reduced or amortized.
BuiltInFunctionAndTypeManagertakes >700ms to initialize for every worker. Each worker initializes all types and compiles all built in functions every startup. It would be nice to turn this into a singleton so that each worker doesn't have to initialize this.
- It's common to initalize TPC-H/DS tables. Generally this is done serially. It can potentially be sped up by copying the tables concurrently.
- I noticed during profiling that one piece that takes a long time to initialize is the static
io.airlift.tpch.TextPool.getDefaultTestPool(). This generates a bunch of random text which is cached when generating tpch tables for the connector. The initialization of this component could be done proactively during server startup to improve TPC-H table initialization - Lastly (this is MacOS specific), make sure that
/etc/hostscontains a hostname other thanlocalhostfor the127.0.0.1and::1entries. This is due to some initialization logic in netty which hangs while looking up DNS names for the machine.
127.0.0.1 localhost <YOUR HOSTNAME>
255.255.255.255 broadcasthost
::1 localhost <YOUR HOSTNAME>
With changes 1, 3, 4, and 5 I was able to get my local environment to launch test clusters in 7-8s. Previously it was 15s. About a 50% improvement. 4s was from the DNS fix, ~2s from parallelizing worker launch. ~1s for parallelizing table copies, ~1s for async init of TPC-H text pool
Presto Component, Service, or Connector
DistributedQueryRunner
Possible Implementation
See #22212
Example Screenshots (if appropriate):
Context
Bothered at waiting so long for some query runners to start up