Support for UDAF
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 30/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- scala
- Domain
- data-engineering
Research direction
Start with the linked Spark Scala documentation for UserDefinedAggregateFunction and Aggregator, then review Spark's DatasetAggregatorSuite examples. Define the scope for supporting UDAFs and the todo item for a TypedAggregator returning a TypedColumn; done should include the intended type-safe aggregation support and corresponding tests.
Written by the indexing model from the issue text.
Description
Currently we have support for UDF only. UDAF are really useful and it will be nice if we can support them.
Relevant scala docs and this. The second seems to be the preferred way to do this with Datasets.
The Aggregator:
trait Aggregator[IN,BUF,OUT]
- IN The input type for the aggregation.
- BUF The type of the intermediate value of the reduction.
- OUT The type of the final output result.
Example from SO:
import org.apache.spark.sql.expressions.Aggregator
import org.apache.spark.sql.{Encoder, Encoders}
class BelowThreshold[I](f: I => Boolean) extends Aggregator[I, Boolean, Boolean]
with Serializable {
val zero = false
def reduce(acc: Boolean, x: I) = acc | f(x)
def merge(acc1: Boolean, acc2: Boolean) = acc1 | acc2
def finish(acc: Boolean) = acc
def bufferEncoder: Encoder[Boolean] = Encoders.scalaBoolean
def outputEncoder: Encoder[Boolean] = Encoders.scalaBoolean
}
val belowThreshold = new BelowThreshold[(String, Int)](_._2 < - 40).toColumn
df.as[(String, Int)].groupByKey(_._1).agg(belowThreshold)
Examples from spark
Other example from Databricks:
val simpleSum = new Aggregator[Int, Int, Int] with Serializable {
def zero: Int = 0 // The initial value.
def reduce(b: Int, a: Int) = b + a // Add an element to the running total
def merge(b1: Int, b2: Int) = b1 + b2 // Merge intermediate values.
def finish(b: Int) = b // Return the final result.
}.toColumn
val ds = Seq(1, 2, 3, 4).toDS()
ds.select(simpleSum).collect
todo: Create an TypedAggregator that returns a TypedColumn.
The entire construct seems to be quite type safe. This could be really simple.
- Dominant language
- Scala
- Stars
- 895
- Forks
- 135
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 3
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from typelevel/frameless
-
Scala 3 builds Openenhancement feature help wanted
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
-
enhancement feature
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
Difficulty 5/5 Over a week Newbie friendliness 30/100
All issues in typelevel/frameless
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
typelevel/sbt-typelevel#929 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
apache/pekko-projection#635 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Area: Excel support
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
orbeon/orbeon-forms#7893 ·
-
x:action/fix x:module/practice-exercise x:size/small
Difficulty 2/5 1-3 hours Newbie friendliness 75/100