metabase/metabase

Sync metadata has n+1 on some drivers

Aperta

#37.742 aperta il 16 gen 2024

 (6 commenti) (0 reazioni) (1 assegnatario)Clojure (6465 fork)batch import
.Backend.Performance.Team/GraphyAdministration/Metadata & SyncPriority:P2Type:Buggood first issue

Metriche repository

Star
 (47.314 stelle)
Metriche merge PR
 (Merge medio 1g 15h) (1000 PR mergiate in 30 g)

Descrizione

During the sync process, we have the method driver/describe-database to find all the tables within a database. For some drivers we implemented so that we only sync tables that the current connection has select privilege against.

The current checking method is executing a query select true from table limit 1 see (metabase.driver.sql-jdbc.sync.interface/have-select-privilege?), this is inefficient and is a basic N+1 problem.

We have fixed this for Postgres, redshift, and MySQL in #37439 by using the output of driver/current-user-table-privileges to filter out tables where we don't have select privileges.

This is a much better approach, so we should extend it to supports: sparksql, oracle, snowflake, athena, sqlite, maraidb, sqlserver. For these we can implement driver/current-user-table-privileges and they will automatically be fast.

Additionally, we have some drivers that do not check select privileges during sync, like bigquery, vertica, or presto; we should update so that those drivers doing this check, too.

Guida contributor