nteract/papermill

bug using Scala jupyter notebook + spylon-kernel

Open

#662 建立於 2022年5月5日

在 GitHub 查看
 (2 留言) (0 反應) (0 負責人)Python (402 fork)batch import
bughelp wanted

倉庫指標

Star
 (5,381 star)
PR 合併指標
 (30 天內沒有已合併 PR)

描述

This issue appears whenever I try to run Scala notebooks (spylon-kernel) notebooks using papermill.

If I make the scala val assignments in a standalone parameters cell, papermill does not inject them properly.

This spylon-kernel notebook example does not change parameters:

%%init_spark
launcher.conf.set("spark.driver.cores", "2")
launcher.conf.set("spark.driver.memory", "8G")
launcher.conf.set("spark.executor.cores", "4")
// this is tagged as `parameters` cell
val a = "this is a variable"
spark.sql(s"select 1 as ${a}").show

This works (if I make the variable call inside of %%init_spark and as a python code). In the output notebook, papermill converted the python assignments into Scala code (!):

// this is tagged as `parameters` cell
%%init_spark
a = "this is a variable"

launcher.conf.set("spark.driver.cores", "2")
launcher.conf.set("spark.driver.memory", "8G")
launcher.conf.set("spark.executor.cores", "4")
spark.sql(s"select 1 as ${a}").show

Note that in this second case, I can't make a Scala variable explicit in a cell.

Relevant versions:

pyspark==3.1.1
papermill==2.3.4
spylon==0.3.0
spylon-kernel==0.4.1
jupyter-client==7.0.6
jupyter-core==4.9.1
jupyter-packaging==0.11.0
jupyter-server==1.11.1
jupyterlab==3.0.16
jupyterlab-pygments==0.1.2
jupyterlab-server==2.8.2
jupyterlab-sparkmonitor==4.1.0

bash script:

#!/bin/bash

papermill \
/home/jovyan/spylon-test.ipynb \
/home/jovyan/spylon-test-output.ipynb \
-p a "ccccc2a"

Is it possible to correct this so it works like it is expected (as the first example)?

Thanks guys, I love papermill, big appreciation for what you are doing for the community.

貢獻者指南