nteract/papermill

bug using Scala jupyter notebook + spylon-kernel

Open

#662 opened on May 5, 2022

View on GitHub
 (2 comments) (0 reactions) (0 assignees)Python (402 forks)batch import
bughelp wanted

Repository metrics

Stars
 (5,381 stars)
PR merge metrics
 (No merged PRs in 30d)

Description

This issue appears whenever I try to run Scala notebooks (spylon-kernel) notebooks using papermill.

If I make the scala val assignments in a standalone parameters cell, papermill does not inject them properly.

This spylon-kernel notebook example does not change parameters:

%%init_spark
launcher.conf.set("spark.driver.cores", "2")
launcher.conf.set("spark.driver.memory", "8G")
launcher.conf.set("spark.executor.cores", "4")
// this is tagged as `parameters` cell
val a = "this is a variable"
spark.sql(s"select 1 as ${a}").show

This works (if I make the variable call inside of %%init_spark and as a python code). In the output notebook, papermill converted the python assignments into Scala code (!):

// this is tagged as `parameters` cell
%%init_spark
a = "this is a variable"

launcher.conf.set("spark.driver.cores", "2")
launcher.conf.set("spark.driver.memory", "8G")
launcher.conf.set("spark.executor.cores", "4")
spark.sql(s"select 1 as ${a}").show

Note that in this second case, I can't make a Scala variable explicit in a cell.

Relevant versions:

pyspark==3.1.1
papermill==2.3.4
spylon==0.3.0
spylon-kernel==0.4.1
jupyter-client==7.0.6
jupyter-core==4.9.1
jupyter-packaging==0.11.0
jupyter-server==1.11.1
jupyterlab==3.0.16
jupyterlab-pygments==0.1.2
jupyterlab-server==2.8.2
jupyterlab-sparkmonitor==4.1.0

bash script:

#!/bin/bash

papermill \
/home/jovyan/spylon-test.ipynb \
/home/jovyan/spylon-test-output.ipynb \
-p a "ccccc2a"

Is it possible to correct this so it works like it is expected (as the first example)?

Thanks guys, I love papermill, big appreciation for what you are doing for the community.

Contributor guide