Convert Spark Dataframe to Dataset with MapStruct
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- java, spark
- Domain
- data-engineering
Research direction
Start by reviewing the mapstruct-examples repository and the linked Stack Overflow question to understand the requested Spark DataFrame-to-Dataset conversion. Identify where a Spark and Java example would fit, then determine whether MapStruct can handle differing database column names and POJO field types. Done means a documented, working example covering that conversion.
Written by the indexing model from the issue text.
Description
When using Apache Spark with Java there is a pretty common use case of converting Spark's Dataframes to POJO-based Datasets. The thing is that many times your Dataframe is imported from a database in which the column namings and types are different from your POJO.
Example for this can be found on the following Stackoverflow question (which has been solved programmatically with a map and rename functions). I was wondering if that example could be solved with MapStruct (and maybe would be added to this examples repo).
- Dominant language
- Java
- Stars
- 1.4k
- Forks
- 504
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from mapstruct/mapstruct-examples
-
Java 21 support Open
Difficulty 4/5 3-5 days Newbie friendliness 35/100
mapstruct/mapstruct-examples#153 · 1 comment ·
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
mapstruct/mapstruct-examples#149 · 1 comment ·
-
mapstruct-protobuf3 use lombok:UnsupportedOperationException(Collections$UnmodifiableCollection) Open
Difficulty 2/5 1-3 hours Newbie friendliness 45/100
mapstruct/mapstruct-examples#142 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 15/100
mapstruct/mapstruct-examples#138 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
mapstruct/mapstruct-examples#127 ·
All issues in mapstruct/mapstruct-examples
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
elastic/gradle-plugins#157 ·
-
enhancement Tools
Difficulty 1/5 Under an hour Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
apache/rocketmq-dashboard#5008 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
DETECT_PARAMETER_NAMES=false silently disables @ConstructorProperties-based Creator detection too Open
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
FasterXML/jackson-databind#6229 ·