metabase/metabase

Sync metadata has n+1 on some drivers

Ouverte

#37 742 ouverte le 16 janv. 2024

 (6 commentaires) (0 réaction) (1 personne assignée)Clojure (6 465 forks)batch import
.Backend.Performance.Team/GraphyAdministration/Metadata & SyncPriority:P2Type:Buggood first issue

Métriques du dépôt

Stars
 (47 314 étoiles)
Métriques de merge PR
 (Merge moyen 1j 15h) (1 000 PRs mergées en 30 j)

Description

During the sync process, we have the method driver/describe-database to find all the tables within a database. For some drivers we implemented so that we only sync tables that the current connection has select privilege against.

The current checking method is executing a query select true from table limit 1 see (metabase.driver.sql-jdbc.sync.interface/have-select-privilege?), this is inefficient and is a basic N+1 problem.

We have fixed this for Postgres, redshift, and MySQL in #37439 by using the output of driver/current-user-table-privileges to filter out tables where we don't have select privileges.

This is a much better approach, so we should extend it to supports: sparksql, oracle, snowflake, athena, sqlite, maraidb, sqlserver. For these we can implement driver/current-user-table-privileges and they will automatically be fast.

Additionally, we have some drivers that do not check select privileges during sync, like bigquery, vertica, or presto; we should update so that those drivers doing this check, too.

Guide contributeur