Decouple parquet-hadoop module from hadoop-mapreduce-client-core
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 68/100
- Issue 类型
- 重构
- 描述清晰度
- 描述清楚
- 活跃度
- 活跃
- 技术栈
- hadoop, java
调研方向
从 parquet-hadoop 模块中的 ParquetReadOptions 和 ParquetInputFormat 开始,然后跟踪现有的常量和过滤器解析逻辑。引入所请求的 org.apache.parquet.conf 类,同时保留现有的 ParquetInputFormat 入口点和兼容性。验证新类和 ParquetReadOptions 不会初始化 FileInputFormat,然后运行 ./mvnw test。
由索引模型根据 Issue 内容生成。
描述
Describe the enhancement requested
Decouple ParquetReadOptions from the legacy Hadoop ParquetInputFormat/FileInputFormat classes, which currently force pulling in the hadoop-mapreduce-client-core dependency (and its transitive JARs).
Summary
To instantiate a org.apache.parquet.hadoop.ParquetReader we need to use org.apache.parquet.ParquetReadOptions, which references a set of keys located in org.apache.parquet.hadoop.ParquetInputFormat and calls a static getFilter method declared also on ParquetInputFormat, which extends org.apache.hadoop.mapreduce.lib.input.FileInputFormat.
ParquetInputFormat is part of the parquet-hadoop module, while FileInputFormat is declared in the hadoop-mapreduce-client-core JAR from the Hadoop project.
Because ParquetReader (via ParquetReadOptions) needs to call ParquetInputFormat.getFilter(...), the JVM is forced to initialize ParquetInputFormat, and initializing a class triggers the loading and initialization of its superclass (FileInputFormat) along with its entire transitive dependency graph (org.apache.hadoop.mapreduce.*). For code that only needs to read a plain Parquet file (and not the MapReduce input-format machinery), this pulls an unwanted, heavy Hadoop-mapreduce dependency into the classpath and link set. AvroParquetReader and ProtoParquetReader extend from ParquetReader and have the same issue.
ParquetInputFormat has three responsibilities:
- define a set of property keys as constants
- deserialize the filter predicates from a configuration value
- support the integration of Parquet files into Hadoop MapReduce
This issue proposes to extract the first two responsibilities into two new Hadoop-agnostic classes, in the org.apache.parquet.conf package:
ParquetInputProperties: the property-key constants onlyParquetInputFilters: the filter deserialization logic
so that the configuration can be used without ever loadingFileInputFormatand its transitive dependencies.
ParquetReader ← org.apache.parquet.hadoop (parquet-hadoop)
│ uses
▼
ParquetReadOptions ← org.apache.parquet (parquet-hadoop)
│ (static import keys + getFilter call)
▼
ParquetInputFormat.getFilter(...) ← org.apache.parquet.hadoop (parquet-hadoop)
│ extends
▼
FileInputFormat<Void, T> ← org.apache.hadoop.mapreduce.lib.input (hadoop-mapreduce-client-core)
│ extends
▼
InputFormat<K, V> ← org.apache.hadoop.mapreduce.lib.input (hadoop-mapreduce-client-core)
│ (transitive)
▼
{ InputSplit, JobContext, TaskAttemptContext,
RecordReader, ... } ← org.apache.hadoop.mapreduce* (hadoop-mapreduce-client-core)
With the new ParquetInputProperties / ParquetInputFilters, building ParquetReadOptions no longer touches any org.apache.hadoop.mapreduce type, so consumers not related to Hadoop avoid transitively including the MapReduce dependency.
Public API / Behavioral change
No behavior change. This is a refactoring:
- New classes
org.apache.parquet.conf.ParquetInputPropertiesand
org.apache.parquet.conf.ParquetInputFilters(inparquet-hadoop) hold the constants and the
ParquetConfiguration-based filter resolution respectively. - The legacy
org.apache.hadoop.conf.Configuration-based entry points remain on
ParquetInputFormat(they are used only via the legacy MapReduce path) and are kept for binary /
source compatibility. - All pre-existing constants on
ParquetInputFormatare now@Deprecatedand delegate to the new
classes; source and binary compatibility are preserved.
Acceptance criteria
ParquetReadOptions(used for plain file reads) references onlyParquetInputProperties/ParquetInputFilters, neverParquetInputFormat/org.apache.hadoop.mapreduce.InputFormat.- Loading
ParquetInputProperties/ParquetInputFilters/ParquetReadOptionsdoes not initializeFileInputFormat. - All existing tests still pass (
./mvnw test).
Component(s)
Core
- 主要语言
- Java
- 星标
- 3.1k
- 派生
- 1.6k
- 平均合并
- 4 天 12 小时
- 30 天内合并 PR
- 28
贡献指南
这个仓库没有索引到贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
apache/parquet-java 的其他 Issue
-
Type: bug
难度 2/5 1-3 小时 新手友好度 68/100
apache/parquet-java#3792 ·
-
难度 2/5 1-3 小时 新手友好度 82/100
apache/parquet-java#3767 ·
-
难度 2/5 1-3 小时 新手友好度 72/100
apache/parquet-java#3695 · 1 条评论 ·
-
难度 2/5 1-3 小时 新手友好度 78/100
apache/parquet-java#3667 ·
-
Type: bug
难度 2/5 1-3 小时 新手友好度 76/100
apache/parquet-java#3587 ·
查看 apache/parquet-java 的全部 Issue
相似的 Issue
-
awaiting triage bug Causes friction Hop Gui P1 P2 Transforms
难度 2/5 1-3 小时 新手友好度 75/100
-
难度 2/5 1-3 小时 新手友好度 75/100
apache/flink-agents#1152 ·
-
难度 2/5 1-3 小时 新手友好度 75/100
-
难度 2/5 1-3 小时 新手友好度 70/100
jenkinsci/blueocean-plugin#5417 ·
-
难度 2/5 1-3 小时 新手友好度 75/100
objectionary/eo-graphs#75 ·