ropensci/skimr

skimr against a database

Open

#358 opened on Dec 2, 2018

View on GitHub
 (10 comments) (1 reaction) (0 assignees)HTML (1,061 stars) (77 forks)batch import
enhancementhelp wantedskim 3

Description

Hey kind skimr folks. For the last year or so I've been pondering how nice it would be to run skim against my big ol' Redshift database. I suspect I'm not the only one who's thought about this. The general idea would be to have as much code as possible execute on the database using the magic of dbplyr.

Today I decided to spend some time trying to understand the skimr code base and think about what might be necessary in order to refactor the code into functions that can play with the limited subset of functions that dbplyr can execute in SQL.

I've done a quick pass through the code and before I started really digging into this I wanted to see if any of you wise folks had given this thought or maybe seen something else implemented elsewhere. It seems like a database friendly skimr would provide a lot of value, it's not a trivial exercise to refactor skimr.

Any input you all have would be much appreciated.

Contributor guide

Tech stack
rsql
Domain
database
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
needs investigation
Prerequisites
R programmingdbplyr packageSQLskimr internals
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
15
Research direction
Investigate the skimr codebase to identify functions that can be refactored to work with dbplyr's limited SQL translation. Focus on understanding how dbplyr works and which skimr operations can be expressed in SQL. The issue mentions that a complete refactoring is non trivial, so start by exploring the current code structure and consider implementing support for a single database backend as a proof of concept.