njtierney/naniar

introduce functions to work with spark

Open

#40 opened on Jan 5, 2017

View on GitHub
 (3 comments) (0 reactions) (0 assignees)R (557 stars) (43 forks)batch import
Ideaenhancementhelp wanted

Description

@MilesMcBain mentioned that it is difficult to work with missing data within spark.

For example:

library(sparklyr)
library(tibble)
library(dplyr)

dat <- tribble(
    ~A, ~B, ~C,
    NA,  1,  1,
    1,  1, NA,
    NA, NA,  1,
    NA, NA, NA,
    1,  1,  1 
)

sc <- spark_connect(master = "local")
spark_dat <- copy_to(sc, dat)

#A crappy non-scalable way to do complete.cases
complete_cases <- 
    spark_dat %>% 
    filter(!is.na(A) & !is.na(B) & !is.na(C)) %>%
    collect()

#A crappy non-scalable way to do find rows with any na 
any_na <-
    spark_dat %>%
    filter(!(!is.na(A) & !is.na(B) & !is.na(C))) %>%
    collect()

It would be great to have naniar functions that also worked with spark.

Not sure how much work this would involve, but it looks like rstudio have a pretty nice extension API.

Just a thought for now, there's a lot of other things that I want to finish up first, but this should be on my roadmap towards version 1.0.0

Contributor guide

Tech stack
None
Domain
data
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
R programmingfamiliarity with naniarknowledge of missing data handlingsparklyr basics
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
25
Research direction
The issue requests creating naniar functions compatible with Spark, as discussed in the issue body with example code using sparklyr. To begin, review the sparklyr extension API at http://spark.rstudio.com/extensions.html. Check the naniar repository for existing infrastructure and consider how functions like 'any na' or 'complete cases' could be implemented for Spark data frames. Look for any related comments in the issue or linked discussions that might provide further direction.
introduce functions to work with spark · njtierney/naniar#40 | Good First Issue