nteract/papermill

bug using Scala jupyter notebook + spylon-kernel

Open

#662 opened on 2022年5月5日

GitHub で見る
 (2 comments) (0 reactions) (0 assignees)Python (402 forks)batch import
bughelp wanted

Repository metrics

Stars
 (5,381 stars)
PR merge metrics
 (30d に merged PR はありません)

説明

This issue appears whenever I try to run Scala notebooks (spylon-kernel) notebooks using papermill.

If I make the scala val assignments in a standalone parameters cell, papermill does not inject them properly.

This spylon-kernel notebook example does not change parameters:

%%init_spark
launcher.conf.set("spark.driver.cores", "2")
launcher.conf.set("spark.driver.memory", "8G")
launcher.conf.set("spark.executor.cores", "4")
// this is tagged as `parameters` cell
val a = "this is a variable"
spark.sql(s"select 1 as ${a}").show

This works (if I make the variable call inside of %%init_spark and as a python code). In the output notebook, papermill converted the python assignments into Scala code (!):

// this is tagged as `parameters` cell
%%init_spark
a = "this is a variable"

launcher.conf.set("spark.driver.cores", "2")
launcher.conf.set("spark.driver.memory", "8G")
launcher.conf.set("spark.executor.cores", "4")
spark.sql(s"select 1 as ${a}").show

Note that in this second case, I can't make a Scala variable explicit in a cell.

Relevant versions:

pyspark==3.1.1
papermill==2.3.4
spylon==0.3.0
spylon-kernel==0.4.1
jupyter-client==7.0.6
jupyter-core==4.9.1
jupyter-packaging==0.11.0
jupyter-server==1.11.1
jupyterlab==3.0.16
jupyterlab-pygments==0.1.2
jupyterlab-server==2.8.2
jupyterlab-sparkmonitor==4.1.0

bash script:

#!/bin/bash

papermill \
/home/jovyan/spylon-test.ipynb \
/home/jovyan/spylon-test-output.ipynb \
-p a "ccccc2a"

Is it possible to correct this so it works like it is expected (as the first example)?

Thanks guys, I love papermill, big appreciation for what you are doing for the community.

コントリビューターガイド