HMS 4.x metadata transformer silently rewrites schema locations

Open
#757 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
kubernetes, rust
Domain
backend, databases

Research direction

Start by reproducing the Hive 4.0.0 metastore behavior with Trino, focusing on schema-location reads and tables created through the Hive connector. Inspect how the operator configures metastore.metadata.transformer.class and verify whether disabling the transformer preserves explicit locations and managed-table behavior. Done means the configuration is handled appropriately and both reported failures are resolved or clearly prevented.

Written by the indexing model from the issue text.

Description

type/bug
Affected Stackable version

26.7

Affected Apache Hive version

4.0.0

Current and expected behavior

Hive 4 introduced metastore.metadata.transformer.class and defaults it to org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.

Against a metastore used by Trino, the transformer causes two distinct failures:

  1. Schema locations are silently rewritten on read. A schema created with an explicit location under hive.metastore.warehouse.dir is stored correctly in DBS, but every read through the Thrift API returns <warehouse>/<dbname>.db instead. Trino's Iceberg connector derives new table locations from that value, so tables land in the wrong prefix.
  2. CREATE TABLE through the Trino Hive connector is converted from MANAGED_TABLE to EXTERNAL_TABLE. Trino then refuses to write to its own table with Cannot write to non-managed Hive table, so INSERT INTO is broken for every table created through the Hive connector.
Possible solution

For our use cases, disabling the transformer may be the right choice here.

Workaround

Disable the transformer through the metastore's config overrides. Note the value is a single
space, not an empty string:

spec:
  metastore:
    configOverrides:
      hive-site.xml:
        metastore.metadata.transformer.class: " "
Additional context

No response

Environment

k8s: v1.35.1 - also seen on OpenShift environments

Would you like to work on fixing this bug?

None

Dominant language
Rust
Stars
29
Forks
2
Avg merge
1d 19h
Merged PRs (30d)
12

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from stackabletech/hive-operator

All issues in stackabletech/hive-operator

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.