trinodb/trino

Support ORC decimal type evolution in ORC reader

Offen

#5.128 geöffnet am 10.09.2020

 (5 Kommentare) (0 Reaktionen) (0 zugewiesene Personen)Java (2.678 Forks)batch import
good first issue

Repository-Metriken

Stars
 (9.113 Sterne)
PR-Merge-Metriken
 (PR-Metriken ausstehend)

Beschreibung

I have an external hive partitioned table, when I change the table's structure (mostly modify column precision), presto query will trigger the following errors:

io.prestosql.spi.PrestoException: Failed to read ORC file: /tmp/test_part_tbl/logdate=20200911/000000_0
   at io.prestosql.plugin.hive.orc.OrcPageSource.handleException(OrcPageSource.java:138)
   at io.prestosql.plugin.hive.orc.OrcPageSourceFactory.lambda$createOrcPageSource$10
                          (OrcPageSourceFactory.java:344)
	at io.prestosql.orc.OrcBlockFactory$OrcBlockLoader.load(OrcBlockFactory.java:83)
	at io.prestosql.spi.block.LazyBlock$LazyData.load(LazyBlock.java:381)
	at io.prestosql.spi.block.LazyBlock$LazyData.getFullyLoadedBlock(LazyBlock.java:360)
	at io.prestosql.spi.block.LazyBlock.getLoadedBlock(LazyBlock.java:276)
   at io.prestosql.plugin.hive.orc.OrcPageSource$SourceColumn$MaskingBlockLoader.load(OrcPageSource.java:275)
	at io.prestosql.spi.block.LazyBlock$LazyData.load(LazyBlock.java:381)
	at io.prestosql.spi.block.LazyBlock$LazyData.getFullyLoadedBlock(LazyBlock.java:360)
	at io.prestosql.spi.block.LazyBlock.getLoadedBlock(LazyBlock.java:276)
	at io.prestosql.plugin.hive.HivePageSource$CoercionLazyBlockLoader.load(HivePageSource.java:549)
	at io.prestosql.spi.block.LazyBlock$LazyData.load(LazyBlock.java:381)
	at io.prestosql.spi.block.LazyBlock$LazyData.getFullyLoadedBlock(LazyBlock.java:360)
	at io.prestosql.spi.block.LazyBlock.getLoadedBlock(LazyBlock.java:276)
	at io.prestosql.spi.Page.getLoadedPage(Page.java:273)
	at io.prestosql.operator.TableScanOperator.getOutput(TableScanOperator.java:304)
	at io.prestosql.operator.Driver.processInternal(Driver.java:379)
	at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)
	at io.prestosql.operator.Driver.tryWithLock(Driver.java:675)
	at io.prestosql.operator.Driver.processFor(Driver.java:276)
	at io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1076)
	at io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
	at io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)
	at io.prestosql.$gen.Presto_340____20200902_012116_2.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.lang.IllegalArgumentException: target scale must be larger than source scale
	at io.prestosql.spi.type.Decimals.rescale(Decimals.java:298)
	at io.prestosql.orc.reader.DecimalColumnReader.readShortNotNullBlock(DecimalColumnReader.java:184)
	at io.prestosql.orc.reader.DecimalColumnReader.readNonNullBlock(DecimalColumnReader.java:164)
	at io.prestosql.orc.reader.DecimalColumnReader.readBlock(DecimalColumnReader.java:125)
	at io.prestosql.orc.OrcBlockFactory$OrcBlockLoader.load(OrcBlockFactory.java:76)
	... 24 more

The above error can be repeated by doing the following instructions:

  1. create table and add partitions via beeline
create external table test_part_tbl(id  int, asset decimal(10,2))
partitioned by (logdate string)
stored as orc
location '/tmp/test_part_tbl'
tblproperties('external.table.purge'='true');

alter table test_part_tbl add partition(logdate='20200910');

insert into test_part_tbl partition (logdate='20200910') values(1,10.22);
  1. modify column and add another partition and insert a record
alter table test_part_tbl add partition(logdate='20200911');

alter table test_part_tbl change column asset asset decimal(10,3);
insert into test_part_tbl partition (logdate='20200911') values(1,10.23);
insert into test_part_tbl partition (logdate='20200911') values(1,10.233);
  1. query test_part_tbl via presto
presto> select * from  default.test_part_tbl;

Query 20200910_053546_00098_cczqj, FAILED, 1 node
Splits: 19 total, 1 done (5.26%)
2.16 [1 rows, 310B] [0 rows/s, 144B/s]

Query 20200910_053546_00098_cczqj failed: Failed to read ORC file: 
hdfs://cfzq/tmp/test_part_tbl/logdate=20200911/000000_0
  1. query test_part_tbl via hive
select * from test_part_tbl;
+-------------------+----------------------+------------------------+
| test_part_tbl.id  | test_part_tbl.asset  | test_part_tbl.logdate  |
+-------------------+----------------------+------------------------+
| 1                 | 10.220               | 20200910               |
| 1                 | 10.230               | 20200911               |
| 1                 | 10.233               | 20200911               |
+-------------------+----------------------+------------------------+
3 rows selected (0.487 seconds)

If I drop partition 20200911 and add it again , query will result all data.

I use Hortonworks HDP 3.1.4.0-315 version

  • Hadoop HDFS: 3.1.1.3.1
  • Hive: 3.1.0
  • OS: CentOS 7.7.1908
  • PrestoSQL: 340

Contributor Guide