nteract/papermill

Run parameter cell as `%%local` for sparkmagic kernels

開放

#652 建立於 2022年3月2日

 (1 則留言) (0 個反應) (0 位負責人)Python (402 個分叉)batch import
enhancementhelp wanted

倉庫指標

星標
 (5,381 顆星)
PR 合併指標
 (30 天內沒有已合併 PR)

描述

🚀 Feature

Allow setting the execution context for the parameter cell (through the use of %% magic commands)

Motivation

Currently, the way that papermill injects the parameter cell it will get executed by the "default" kernel of the notebook which when executing a sparkmagic kernel notebook is equivalent to using the %%spark magic.

However, we often need to perform some initial setup, for example creating and connecting the cluster that the notebook will interact with, which is done by executing %%local cells, then connecting to the cluster and executing the %%spark cells.

Without the ability to parse the parameters in the local executor, in our use-case there is no Spark cluster to connect to when the parameter cell is executed, leading to a failed execution.

So we'd like a way to annotate the injected parameter cell to run in %%local context, so we can set up and connect to a Spark cluster using local (e.g. boto3) commands, and execute the %%spark cells later.

貢獻者指南