kernel_name is set, but `which python` and `which pip` point to the wrong paths

🐛 Bug

I'm not sure if this is expected or not. Our use case at Google is that we are testing multiple notebooks by executing them. We dynamically create kernels and create environments for each. This is required to test the notebooks independently.

    # Create environment
    kernel_name = str(uuid.uuid4())
    env_name = f"{ENVIRONMENTS_PATH}/{kernel_name}"
    virtualenv.cli_run([env_name, "--system-site-packages"])

    # Create kernel spec
    kernel_spec = {
        "argv": [
            f"{env_name}/bin/python",
            "-m",
            "ipykernel_launcher",
            "-f",
            "{connection_file}",
        ],
        "display_name": "Python 3",
        "language": "python",
    }
    kernel_spec_folder = os.path.join(KERNELS_SPECS_PATH, kernel_name)
    kernel_spec_file = os.path.join(kernel_spec_folder, "kernel.json")

    # Create kernel spec folder
    if not os.path.exists(os.path.dirname(kernel_spec_file)):
        try:
            os.makedirs(os.path.dirname(kernel_spec_file))
        except OSError as exc:  # Guard against race condition
            if exc.errno != errno.EEXIST:
                raise

    with open(kernel_spec_file, mode="w", encoding="utf-8") as f:
        json.dump(kernel_spec, f)

    # Install kernel
    kernel_spec_manager = KernelSpecManager()
    kernel_spec_manager.install_kernel_spec(
        source_dir=kernel_spec_folder, kernel_name=kernel_name
    )

Then we run the kernel using papermill by passing the kernel_name var to the Papermill.execute_notebook method.

I created a test notebook with a cell that has ! which pip and ! which python. I expected it to point to the pip and python in the environment folder that I created earlier, but it always points to the pip and python that I run the parent script (the one coordinating the tests) in.

Reference: https://github.com/GoogleCloudPlatform/ai-platform-samples/pull/439