Add support for TypedDeltaTable
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- scala, spark
- Domain
- data-engineering
Research direction
Start by reviewing Frameless's existing TypedDataset API and delta.io's DeltaTable API, then determine how a new frameless-delta module would expose typed merge actions. Use the proposed TypedDeltaTable usage as the target shape; done means the module supports strongly typed merge operations over DeltaTable.
Written by the indexing model from the issue text.
Description
Frameless introduced the TypedDataset on top of Apache Spark's Dataset API. The aim of this ticket is to create a TypedDeltaTable on top of delta.io's DeltaTable API. This would allow to have strongly typed merge actions, something that would look like the following:
val spark: SparkSession = ...
val updates: TypedDataset[U] = ...
val target: TypedDeltaTable[T] = TypedDataTable.unsafeForPath[T](spark, path)
val sq = target
.merge[U](updates, target('id) === updates('id))
.whenMatched(target('someSymbol) != updates('anotherSymbol))
.update(target('someSymbol) := updates('anotherSymbol))
.execute()
sq.awaitTermination()
This could be included in a new module, called frameless-delta.
- Dominant language
- Scala
- Stars
- 895
- Forks
- 135
- Avg merge
- 7h 38m
- Merged PRs (30d)
- 1
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from typelevel/frameless
-
Scala 3 buildsOpenenhancement feature help wanted
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
-
enhancement feature
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
Difficulty 5/5 Over a week Newbie friendliness 30/100
All issues in typelevel/frameless
Similar issues
-
arrays_zip with two same-named inputs fails with "ArrowArray struct has 2 children (expected 1)"Openbug requires-triage
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/datafusion-comet#6251 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
lichess-org/lila#21826 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
ergoplatform/ergo#2579 ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
apache/texera#8682 · 4 comments ·
Maintainers usually reply within 1 day
-
x:action/improve x:size/tiny x:type/content
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day