Dots in column names raises assertion error in sqlachemy hive

Open
#413 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
68/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
python, sqlalchemy
Domain
databases

Research direction

Start in pyhive/sqlalchemy_hive.py at HiveCompiler.visit_column, then reproduce the failure with the quoted dotted column examples from the issue. Check how the compiled identifier is parsed and verify that columns such as col.id.part compile successfully while retaining the table and database qualification shown in the examples.

Written by the indexing model from the issue text.

Description

Dots in column names get counted as separators between db name, table name and column name.

Reason is code

dot_count = result.count('.')
assert dot_count in (0, 1, 2), "Unexpected visit_column result {}".format(result)

do not take care about quotes. See pyhive.sqlalchemy_hive.HiveCompiler.visit_column

In order to reproduce column should have two dots table.`col.id.part`, or one dot plus explicit dbname prefix dbname.table.`col.id`

from databricks_dbapi.sqlalchemy_dialects.base import DatabricksDialectBase
from sqlalchemy.ext.declarative import declarative_base

from sqlalchemy import Column, Integer
from sqlalchemy.orm import Query

Base = declarative_base()


class Test(Base):
    __tablename__ = 'test'
    id = Column(Integer, name='col.id.part', primary_key=True)


q = Query(Test)
# SELECT test.`col.id.part` FROM test
q.statement.compile(dialect=DatabricksDialectBase())
Traceback (most recent call last):
  File "/home/kotofos/.config/JetBrains/PyCharm2021.2/scratches/scratch_1.py", line 16, in <module>
    q.statement.compile(dialect=DatabricksDialectBase())
  File "<string>", line 1, in <lambda>
  File "/home/kotofos/Documents/slashdb/slashdb_venv/lib64/python3.6/site-packages/sqlalchemy/sql/elements.py", line 481, in compile
    return self._compiler(dialect, bind=bind, **kw)
  File "/home/kotofos/Documents/slashdb/slashdb_venv/lib64/python3.6/site-packages/sqlalchemy/sql/elements.py", line 487, in _compiler
    return dialect.statement_compiler(dialect, self, **kw)
  File "/home/kotofos/Documents/slashdb/slashdb_venv/lib64/python3.6/site-packages/sqlalchemy/sql/compiler.py", line 592, in __init__
    Compiled.__init__(self, dialect, statement, **kwargs)
  File "/home/kotofos/Documents/slashdb/slashdb_venv/lib64/python3.6/site-packages/sqlalchemy/sql/compiler.py", line 322, in __init__
    self.string = self.process(self.statement, **compile_kwargs)
  File "/home/kotofos/Documents/slashdb/slashdb_venv/lib64/python3.6/site-packages/sqlalchemy/sql/compiler.py", line 352, in process
    return obj._compiler_dispatch(self, **kwargs)
  File "/home/kotofos/Documents/slashdb/slashdb_venv/lib64/python3.6/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
    return meth(self, **kw)
  File "/home/kotofos/Documents/slashdb/slashdb_venv/lib64/python3.6/site-packages/sqlalchemy/sql/compiler.py", line 2177, in visit_select
    for name, column in select._columns_plus_names
  File "/home/kotofos/Documents/slashdb/slashdb_venv/lib64/python3.6/site-packages/sqlalchemy/sql/compiler.py", line 2177, in <listcomp>
    for name, column in select._columns_plus_names
  File "/home/kotofos/Documents/slashdb/slashdb_venv/lib64/python3.6/site-packages/sqlalchemy/sql/compiler.py", line 1929, in _label_select_column
    return result_expr._compiler_dispatch(self, **column_clause_args)
  File "/home/kotofos/Documents/slashdb/slashdb_venv/lib64/python3.6/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
    return meth(self, **kw)
  File "/home/kotofos/Documents/slashdb/slashdb_venv/lib64/python3.6/site-packages/pyhive/sqlalchemy_hive.py", line 161, in visit_column
    assert dot_count in (0, 1, 2), "Unexpected visit_column result {}".format(result)
AssertionError: Unexpected visit_column result `test`.`col.id.part`
Dominant language
Python
Stars
1.7k
Forks
545
PR merge metrics
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from dropbox/PyHive

All issues in dropbox/PyHive

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.