Repository metrics
- Stars
- (5,381 stars)
- PR merge metrics
- (No merged PRs in 30d)
Description
This issue appears whenever I try to run Scala notebooks (spylon-kernel) notebooks using papermill.
If I make the scala val assignments in a standalone parameters cell, papermill does not inject them properly.
This spylon-kernel notebook example does not change parameters:
%%init_spark
launcher.conf.set("spark.driver.cores", "2")
launcher.conf.set("spark.driver.memory", "8G")
launcher.conf.set("spark.executor.cores", "4")
// this is tagged as `parameters` cell
val a = "this is a variable"
spark.sql(s"select 1 as ${a}").show
This works (if I make the variable call inside of %%init_spark and as a python code). In the output notebook, papermill converted the python assignments into Scala code (!):
// this is tagged as `parameters` cell
%%init_spark
a = "this is a variable"
launcher.conf.set("spark.driver.cores", "2")
launcher.conf.set("spark.driver.memory", "8G")
launcher.conf.set("spark.executor.cores", "4")
spark.sql(s"select 1 as ${a}").show
Note that in this second case, I can't make a Scala variable explicit in a cell.
Relevant versions:
pyspark==3.1.1
papermill==2.3.4
spylon==0.3.0
spylon-kernel==0.4.1
jupyter-client==7.0.6
jupyter-core==4.9.1
jupyter-packaging==0.11.0
jupyter-server==1.11.1
jupyterlab==3.0.16
jupyterlab-pygments==0.1.2
jupyterlab-server==2.8.2
jupyterlab-sparkmonitor==4.1.0
bash script:
#!/bin/bash
papermill \
/home/jovyan/spylon-test.ipynb \
/home/jovyan/spylon-test-output.ipynb \
-p a "ccccc2a"
Is it possible to correct this so it works like it is expected (as the first example)?
Thanks guys, I love papermill, big appreciation for what you are doing for the community.