prestodb/presto

Improve DistributedQueryRunner startup time

Open

#22.213 aperta il 14 mar 2024

Vedi su GitHub
 (1 commento) (1 reazione) (1 assegnatario)Java (5240 fork)batch import
good first issueperformance-improvement

Metriche repository

Star
 (15.558 star)
Metriche merge PR
 (Merge medio 34g 14h) (120 PR mergiate in 30 g)

Descrizione

Expected Behavior or Use Case

Our testing infrastructure makes heavy use of the DistributedQueryRunner class such as in the IcebergQueryRunner and HiveQueryRunner. The startup time of the query runner contributes a respectable portion of the testing time. I think there are some optimizations that can be made.

  1. I noticed that everything is started serially. We could try to improve the overall startup time by parallelizing the launching of the workers and coordinator once the discovery server starts.
  2. There are other bits during startup which contribute a significant amount of time that could ideally be reduced or amortized.
  • BuiltInFunctionAndTypeManager takes >700ms to initialize for every worker. Each worker initializes all types and compiles all built in functions every startup. It would be nice to turn this into a singleton so that each worker doesn't have to initialize this.
  1. It's common to initalize TPC-H/DS tables. Generally this is done serially. It can potentially be sped up by copying the tables concurrently.
  2. I noticed during profiling that one piece that takes a long time to initialize is the static io.airlift.tpch.TextPool.getDefaultTestPool(). This generates a bunch of random text which is cached when generating tpch tables for the connector. The initialization of this component could be done proactively during server startup to improve TPC-H table initialization
  3. Lastly (this is MacOS specific), make sure that /etc/hosts contains a hostname other than localhost for the 127.0.0.1 and ::1 entries. This is due to some initialization logic in netty which hangs while looking up DNS names for the machine.
127.0.0.1       localhost <YOUR HOSTNAME>
255.255.255.255 broadcasthost
::1             localhost <YOUR HOSTNAME>

With changes 1, 3, 4, and 5 I was able to get my local environment to launch test clusters in 7-8s. Previously it was 15s. About a 50% improvement. 4s was from the DNS fix, ~2s from parallelizing worker launch. ~1s for parallelizing table copies, ~1s for async init of TPC-H text pool

Presto Component, Service, or Connector

DistributedQueryRunner

Possible Implementation

See #22212

Example Screenshots (if appropriate):

Context

Bothered at waiting so long for some query runners to start up

Guida contributor