cloudpipe/cloudpickle

CI configuration to run the tests of PySpark against development version of cloudpickle

Open

#237 opened on Jan 29, 2019

View on GitHub
 (4 comments) (0 reactions) (0 assignees)Python (1,407 stars) (152 forks)batch import
enhancementhelp wanted

Description

In the recently merged #236, @pierreglaser added the ability to launch the tests of some downstream projects (loky, joblib and dask/distributed) in order to be able to detect whether if a specific PR is likely to introduce a regression in downstream project just by adding the [ci downstream] flag in a commit message of the last commit of the PR.

It would be great to add the build configuration to launch the PySpark test suite in a similar manner.

Contributor guide

Tech stack
pythonspark
Domain
ci cdtesting
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
knowledge of CI configurationbasic understanding of PySpark
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60
Research direction
Review the implementation in #236 that adds downstream testing for loky, joblib, and dask. Identify the CI configuration files (e.g., .travis.yml or similar) and add a similar job for PySpark. Ensure the PySpark test suite can be run against the development version of cloudpickle. Consult maintainers if there are specific PySpark test requirements.