Missing Dataset methods
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- scala, spark
- Domain
- data-engineering
Research direction
Start by reviewing frameless.TypeDataset and comparing its API with the listed Spark Dataset TODO entries. Check the referenced issues for writeStream and withWatermark before choosing a method. Done means the selected missing API methods are implemented and the TODO list accurately reflects their status.
Written by the indexing model from the issue text.
Description
Here is an exhaustive status of the API implemented by frameless.TypeDataset compared to Spark's Dataset. We are getting pretty close to 100% API coverage 😄
Won't fix:
- Dataset alias(String alias) inherently unsafe
- Dataset withColumnRenamed(String existingName, String newName) inherently unsafe
- void createGlobalTempView(String viewName) inherently unsafe
- void createOrReplaceTempView(String viewName) inherently unsafe
- void createTempView(String viewName) inherently unsafe
- void registerTempTable(String tableName) inherently unsafe
- Dataset where(String conditionExpr) use select instead
TODO:
- KeyValueGroupedDataset<K,T> groupByKey(scala.Function1<T,K> func, Encoder evidence3)
- DataFrameNaFunctions na()
- DataFrameStatFunctions stat()
- Dataset dropDuplicates(String col1, String... cols)
- Dataset describe(String... cols)
- DataStreamWriter writeStream() (see #232)
- Dataset withWatermark(String eventTime, String delayThreshold) (see #232)
- RelationalGroupedDataset cube(Column... cols) (WIP #246)
- RelationalGroupedDataset rollup(String col1, String... cols) (WIP #246)
Done:
- Dataset sort(String sortCol, String... sortCols) (#248)
- Dataset sortWithinPartitions(String sortCol, String... sortCols) (#248)
- Dataset repartition(int numPartitions, Column... partitionExprs)
- Dataset drop(String... colNames) (#209)
- Dataset join(Dataset<?> right, Column joinExprs, String joinType)
- Dataset<scala.Tuple2<T,U>> joinWith(Dataset other, Column condition, String joinType)
- Dataset crossJoin(Dataset<?> right)
- Dataset agg(Column expr, Column... exprs)
- Column apply(String colName)
- Dataset as(Encoder evidence2)
- Dataset cache()
- Dataset coalesce(int numPartitions)
- Column col(String colName)
- Object collect()
- long count()
- Dataset distinct()
- Dataset except(Dataset other)
- void explain(boolean extended)
- <A,B> Dataset explode(String inputColumn, String outputColumn, scala.Function1<A,TraversableOnce<B f)
- Dataset filter(Column condition)
- Dataset filter(scala.Function1<T,Object> func)
- T first() (as firstOption)
- Dataset flatMap(scala.Function1<T,TraversableOnce> func, Encoder evidence8)
- void foreach(ForeachFunction func)
- void foreachPartition(scala.Function1<Iterator,scala.runtime.BoxedUnit> f)
- RelationalGroupedDataset groupBy(String col1, String... cols)
- Dataset intersect(Dataset other)
- Dataset limit(int n)
- Dataset map(scala.Function1<T,U> func, Encoder evidence6)
- Dataset mapPartitions(MapPartitionsFunction<T,U> f, Encoder encoder)
- Dataset persist(StorageLevel newLevel)
- void printSchema()
- RDD rdd()
- T reduce(scala.Function2<T,T,T> func) (as reduceOption)
- Dataset repartition(int numPartitions)
- Dataset sample(boolean withReplacement, double fraction, long seed)
- Dataset select(String col, String... cols)
- void show(int numRows, boolean truncate)
- Object take(int n)
- Dataset toDF()
- String toString()
- Dataset transform(scala.Function1<Dataset,Dataset> t)
- Dataset union(Dataset other)
- Dataset unpersist(boolean blocking)
- Dataset withColumn(String colName, Column col)
- Dataset orderBy(String sortCol, String... sortCols)
- String[] columns()
- org.apache.spark.sql.execution.QueryExecution queryExecution()
- StructType schema()
- SparkSession sparkSession()
- SQLContext sqlContext()
- Dataset checkpoint(boolean eager)
- String[] inputFiles()
- boolean isLocal()
- boolean isStreaming()
- Dataset[] randomSplit(double[] weights, long seed)
- StorageLevel storageLevel()
- Dataset toJSON()
- java.util.Iterator toLocalIterator()
- DataFrameWriter write()
- Dominant language
- Scala
- Stars
- 896
- Forks
- 135
- PR merge metrics
- No merged PRs in 30d
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from typelevel/frameless
-
Scala 3 buildsOpenenhancement feature help wanted
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
-
enhancement feature
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
Difficulty 5/5 Over a week Newbie friendliness 30/100
All issues in typelevel/frameless
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
com-lihaoyi/mill#7776 ·
Maintainers usually reply within 1 day
-
Module: Form Builder
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
orbeon/orbeon-forms#7943 ·
Maintainers usually reply within 2 days
-
module: core
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
bigbluebutton/bigbluebutton#26028 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
lambdaworks/zio-elasticsearch#869 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
bazel-contrib/rules_scala#2018 ·
Maintainers usually reply within 2 days