Question: Documentation for supported data types of parameters? (E.g. What is proper way to use tuples with Papermill?)
#586 opened on Apr 6, 2021
Repository metrics
- Stars
- (5,381 stars)
- PR merge metrics
- (No merged PRs in 30d)
Description
MWE:
input_notebook.ipynb has two cells, a parameters cell:
numbers = None
and a code execution cell
small, big = numbers
papermill_notebook.ipynb has one cell with the code:
import papermill as pm
pm.execute_notebook(
'input_notebook.ipynb',
'output_notebook.ipynb',
parameters = {'numbers':(-1,1)}
)
Tuples are a basic, immutable datatype. The parameter is a tuple whose entries are also immutable and even more basic.
However, attempting to run papermill_notebook.ipynb leads to:
PapermillExecutionError:
---------------------------------------------------------------------------
Exception encountered at "In [2]":
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-3-4b9474dadad0> in <module>
----> 1 small, big = numbers
ValueError: too many values to unpack (expected 2)
This is because papermill converted the length two tuple, (-1,1), to the length 7 string "(-1,1)".
This definitely wasn't the behavior I was expecting, which seems to suggest I don't sufficiently understand the "philosophy" of Papermill. Question/Request: Would it be possible to point to a relevant portion of the documentation explaining to users which data types they should and should not expect to be able to pass as parameters?
I haven't actually looked at the papermill source code in detail yet, but I'm guessing it has something to do with JSON serialization, and the fact that there is no notion of 'tuple' in JSON?
(E.g. when I tried to pass a lambda as a parameter once I got a '... not JSON serializable error', so maybe papermill is converting to JSON to avoid security risks associated with pickle or whatever.)
For example the above example works using parameters = {'numbers':[-1,1]} (i.e. a list) instead, presumably because lists correspond to the JSON array objects so are serializable without first converting to string?