`CREATE TABLE` / `DROP TABLE` hang on a current-thread tokio runtime and panic outside a runtime

Open
  • Is something wrong?
#37 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@mikamikasuki is already working on this.

Since Oct 2, 2026.

  • #45 by @mikamikasuki — open

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
rust, sql
Domain
databases

Research direction

Start in crates/datafusion/src/schema.rs at the register_table and deregister_table implementations linked in the issue, then reproduce the behavior with the delayed catalog on a current-thread Tokio runtime and outside any runtime. Trace how the synchronous SchemaProvider methods wait for catalog operations. Done means CREATE TABLE and DROP TABLE complete or return an error on both runtime flavors without panicking outside a runtime.

Written by the indexing model from the issue text.

Description

bug
Describe the bug

IcebergSchemaProvider::register_table and deregister_table are synchronous SchemaProvider methods that DataFusion calls while executing SQL. They run the catalog call inside tokio::task::spawn_blocking with Handle::current().block_on, then wait for the JoinHandle with futures::executor::block_on on the calling thread (schema.rs#L176-L207, schema.rs#L221-L243).

That blocks the thread that called into DataFusion. On a current-thread runtime, that thread is the only one that drives tokio's I/O and timer drivers. Handle::block_on on the blocking-pool thread doesn't drive them, so a catalog call that awaits network I/O or a timer never completes. That covers REST, Glue, HMS and similar catalogs.

To Reproduce
  1. Wrap a MemoryCatalog in a Catalog implementation whose methods await tokio::time::sleep(Duration::from_millis(2)) before delegating, to stand in for a network catalog.
  2. On a runtime built with tokio::runtime::Builder::new_current_thread().enable_all(), register it through IcebergCatalogProvider and run CREATE TABLE ice.ns.created (id INT NOT NULL).

No result after 10 seconds: the statement hangs.

Calling register_table from a thread with no tokio runtime panics:

thread '<unnamed>' panicked at crates/datafusion/src/schema.rs:177:22:
there is no reactor running, must be called from the context of a Tokio 1.x runtime

On a multi-thread runtime the statement completed in this test, but it keeps the calling thread blocked for the whole catalog round trip.

Expected behavior

CREATE TABLE and DROP TABLE complete or return an error on either runtime flavor. The synchronous methods don't panic when called outside a runtime.

Additional context

#24 (section 3) discusses moving remote create/drop behind an async boundary; this issue is the concrete failure in the current implementation. #[tokio::test] uses a current-thread runtime by default. The existing tests pass only because MemoryCatalog on a local filesystem never needs the I/O or timer driver.

Dominant language
Rust
Stars
30
Forks
30
Avg merge
3d 19h
Merged PRs (30d)
15

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/datafusion-iceberg

All issues in apache/datafusion-iceberg

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.