Container Management

Currently funcX defers the responsibility of container fetching to the user. However, we need to have better container management to handle the case where users might dynamically request containers not yet available on the resource.

Here's a sketch of v1 for Container management:

1. User registers a container specifying url and type: c_uuid = register(container_url, type='singularity|docker|shifter')
2. Extra mechanism to manually register additional types against same container id a_bool = register(container_url, type='...', container_uuid=X)
3. When user invokes a function tagged with a container uuid, the service issues the following to the endpoint: func_uuid:container_uuid:<task_buffer>
4. The endpoint maintains a local table mapped to a json file in ~/funcx that contains, the container_uuid: local_container_path mappings. The local mapping is either a container image file on disk or some container url that the provider can resolve.
   • migrate_tasks_to_internal checks the container_uuid, and if it is missing in the table, the endpoint agent has to put the task into an on_hold queue and triggers the container fetch process.
   • If container_uuid is present, the uuid string is replaced by the local path.
5. Endpoint agents query the funcx_service for info on the container_uuid, and try to download the appropriate container to the local disk and update the internal mapping. Once done, triggers a callback, that pops items in the on_hold tasks onto the pending queue with the container_uuid replaced by the local_path.

We want the container fetch to trigger a callback with the container_uuid so that the tasks matching that can be migrated to the pending queue. If instead a failure was encountered, we want an exception object to propagate back.