metadata.reflect() not working causing `DESCRIBE default` because of possible bug in get_table_names()
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 45/100
Research direction
Start in sqlalchemy_hive.py around line 376 and inspect how get_table_names() processes the rows returned by SHOW TABLES. Reproduce the issue with the provided Hive connection and metadata.reflect(); done means reflection lists the tables in the schema without issuing DESCRIBE default.
Written by the indexing model from the issue text.
Description
The following snippet (assuming hive.thriftserver running on 10000 port)
engine = create_engine('hive://localhost:10000/default')
metadata = MetaData(bind=engine)
metadata.reflect()
causes
sqlalchemy.exc.OperationalError: (pyhive.exc.OperationalError) TExecuteStatementResp(status=TStatus(statusCode=3,
...
errorMessage="Error running query: org.apache.spark.sql.AnalysisException: Table or view not found: default; line 1 pos 9;\n'DescribeRelation false, [col_name#7203, data_type#7204, comment#7205]\n+- 'UnresolvedTableOrView [default], DESCRIBE TABLE, true\n"), operationHandle=None)
[SQL: DESCRIBE default]
This is because of possibly incorrect implementation of get_table_names in sqlalchemy_hive.py::376 (v0.7.1.dev0)
def get_table_names(self, connection, schema=None, **kw):
query = 'SHOW TABLES'
...
return [row[0] for row in connection.execute(text(query))]
while in row[0] in spark sql is schema name
SHOW TABLES;
+-----------+------------+--------------+--+
| database | tableName | isTemporary |
+-----------+------------+--------------+--+
| default | sam | false |
| default | sam1 | false |
| default | suj | false |
+-----------+------------+--------------+--+
so proposed fix is
def get_table_names(self, connection, schema=None, **kw):
query = 'SHOW TABLES'
if schema:
query += ' IN ' + self.identifier_preparer.quote_identifier(schema)
return [row[1] for row in connection.execute(text(query))]
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 545
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from dropbox/PyHive
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
Similar issues
-
documentation help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
simonw/sqlite-utils#872 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100