[Bug] information_schema.tables on an external Iceberg catalog loads every table from the remote catalog (unconditional COMMENT fill)

Open Beginner friendly
#65,988 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
78/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
java, sql
Domain
backend, databases

Research direction

Start in FrontendServiceImpl.java at listTableStatus, then read the existing requiredColumns checks around the status-column fills and the TABLE_COMMENT path. Reproduce with SELECT table_schema, table_name FROM information_schema.tables against an external Iceberg REST catalog, and confirm that the projection no longer triggers per-table metadata loads while TABLE_COMMENT queries still return comments.

Written by the indexing model from the issue text.

Description

Search before asking
  • I had searched in the issues and found no similar issues.
Version

master (reproduced on a 2026-07-22 master build)

What's Wrong?

Querying information_schema.tables while switched to an external Iceberg catalog (REST catalog in our case) issues one blocking loadTable against the remote catalog for every table in the catalog, even when the query only projects cheap columns like TABLE_SCHEMA/TABLE_NAME.

On a REST catalog with a few thousand tables, a single listing takes minutes and floods the catalog service with per-table GETs. Since the loads happen inside the listTableStatus thrift handler, clients with statement timeouts (BI tools, dashboards doing schema discovery) just see the query time out. Repeated runs pay the full cost again once the catalog metadata refresh interval passes.

Measured on our cluster: a catalog with 3,882 tables took 2m52s; one with 4,365 tables took 5m38s, with thousands of catalog GETs per scan.

jstack of the master FE during a SELECT table_schema, table_name FROM information_schema.tables scan:

at org.apache.iceberg.rest.RESTSessionCatalog.loadTable(RESTSessionCatalog.java:399)
...
at org.apache.doris.datasource.iceberg.IcebergUtils.getIcebergTable(IcebergUtils.java:943)
at org.apache.doris.datasource.iceberg.IcebergExternalTable.getIcebergTable(IcebergExternalTable.java:150)
at org.apache.doris.datasource.iceberg.IcebergExternalTable.properties(IcebergExternalTable.java:429)
at org.apache.doris.datasource.iceberg.IcebergExternalTable.getComment(IcebergExternalTable.java:155)
at org.apache.doris.service.FrontendServiceImpl.listTableStatus(FrontendServiceImpl.java:756)

Root cause is the interaction of two changes:

  • #64263 made IcebergExternalTable.getComment() read the comment property from the table metadata, which requires loading the table from the remote catalog.
  • #64933 introduced required-columns pruning in listTableStatus, guarding every status column (ENGINE, CREATE_TIME, TABLE_ROWS, ...) — but the status.setComment(table.getComment()) fill remained unconditional.

So the one column whose fill is a remote round-trip is also the one column that is always computed.

What You Expected?

A projection that does not include TABLE_COMMENT should not load table metadata from the remote catalog, the same way it already skips row counts and the other pruned status columns.

How to Reproduce?
  1. Create an Iceberg REST catalog with a large number of tables (a few hundred is enough to see it clearly).
  2. SWITCH to the catalog and run SELECT table_schema, table_name FROM information_schema.tables;
  3. Observe one loadTable request per table on the catalog service, and minutes of latency on large catalogs.
Anything Else?

Fix appears to be a one-hunk change: guard the comment fill with the same needTableStatusColumn(requiredColumns, "TABLE_COMMENT") check used by the other columns. I will attach a PR.

Are you willing to submit PR?
  • Yes I am willing to submit a PR!
Code of Conduct
Dominant language
Java
Stars
16k
Forks
4k
Avg merge
2d 12h
Merged PRs (30d)
569

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/doris

All issues in apache/doris

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.