nteract/papermill

Run parameter cell as `%%local` for sparkmagic kernels

Open

#652 opened on Mar 2, 2022

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Python (402 forks)batch import
enhancementhelp wanted

Repository metrics

Stars
 (5,381 stars)
PR merge metrics
 (No merged PRs in 30d)

Description

🚀 Feature

Allow setting the execution context for the parameter cell (through the use of %% magic commands)

Motivation

Currently, the way that papermill injects the parameter cell it will get executed by the "default" kernel of the notebook which when executing a sparkmagic kernel notebook is equivalent to using the %%spark magic.

However, we often need to perform some initial setup, for example creating and connecting the cluster that the notebook will interact with, which is done by executing %%local cells, then connecting to the cluster and executing the %%spark cells.

Without the ability to parse the parameters in the local executor, in our use-case there is no Spark cluster to connect to when the parameter cell is executed, leading to a failed execution.

So we'd like a way to annotate the injected parameter cell to run in %%local context, so we can set up and connect to a Spark cluster using local (e.g. boto3) commands, and execute the %%spark cells later.

Contributor guide