apache/beam

Deprecate .withProjection from ParquetIO

Open

#20,770 创建于 2022年6月4日

在 GitHub 查看
 (1 评论) (0 反应) (0 负责人)Java (4,097 fork)batch import
P3good first issueimprovementiojavaparquet

仓库指标

Star
 (7,313 star)
PR 合并指标
 (平均合并 6天) (30 天内合并 243 个 PR)

描述

There are multiple issues wrong with the API of withProjection:

  1. The current API requires an extra encoderSchema that is not needed when projecting data in Parquet. The simplest way to get this with the Parquet API is by passing the projectionSchema like this:

{color:#000000}AvroReadSupport{color}.setAvroReadSchema({color:#871094}conf{color}, {color:#871094}projectionSchema{color}); olor:#000000}AvroReadSupport{color}.setRequestedProjection({color:#871094}conf{color}, {color:#871094}projectionSchema{color}); We can offer an alternative method withProjection(Configuration conf, List<String\> fields) so users don't have to build their own projection Schema, but historically we have let users to rely on the upstream connector API. If we follow this we can better document in ParquetIO how to project fields by relying in the Parquet APIs and avoid maintaining this extra code in the Beam side.

 

Imported from Jira BEAM-11908. Original Jira may contain additional context. Reported by: iemejia.

贡献者指南