IpcMode is not handled correctly

Start by tracing how the deployment manifest's HostConfig IpcMode is converted into the generated docker-compose.yml, then compare Docker Compose's ipc service syntax with Docker create options. Check the analogous network_mode conversion mentioned in the report. Done means iotedgehubdev start handles shared IPC references without the nonexistent-container error while preserving valid deployment behavior on a real edge device.

When using a deployment manifest that specifies an IpcMode in hostconfig that accesses shared memory in another container, iotedgehubdev start fails due to a nonexistent container error.

Relevant version numbers:

iotedgehubdev, version 0.14.3
docker-compose version 1.28.5, build c4eb3a1f
docker-py version: 4.4.4
CPython version: 3.7.10
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019

with docker-compose versions prior to 1.27.0 a different error message is thrown.

Sample deployment manifest

{
  "modulesContent": {
    "$edgeAgent": {
      "properties.desired": {
        "schemaVersion": "1.0",
        "runtime": {
          "type": "docker",
          "settings": {
            "minDockerVersion": "v1.25",
            "loggingOptions": "",
            "registryCredentials": {}
          }
        },
        "systemModules": {
          "edgeAgent": {
            "type": "docker",
            "settings": {
              "image": "mcr.microsoft.com/azureiotedge-agent:1.0",
              "createOptions": "{}"
            }
          },
          "edgeHub": {
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "mcr.microsoft.com/azureiotedge-hub:1.0",
              "createOptions": "{\"HostConfig\":{\"PortBindings\":{\"5671/tcp\":[{\"HostPort\":\"5671\"}],\"8883/tcp\":[{\"HostPort\":\"8883\"}],\"443/tcp\":[{\"HostPort\":\"443\"}]}}}"
            }
          }
        },
        "modules": {
          "lvaEdge": {
            "version": "1.0",
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "startupOrder": 1,
            "settings": {
              "image": "mcr.microsoft.com/media/live-video-analytics:2",
              "createOptions": "{\"HostConfig\":{\"LogConfig\":{\"Type\":\"\",\"Config\":{\"max-size\":\"10m\",\"max-file\":\"10\"}},\"Binds\":[\"/tmp/:/var/media/\"],\"IpcMode\":\"shareable\"}}"
            }
          },
          "rtspsim": {
            "version": "1.0",
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "mcr.microsoft.com/lva-utilities/rtspsim-live555:1.2",
              "createOptions": "{\"HostConfig\":{\"IpcMode\":\"container:lvaEdge\"}}"
            }
          }
        }
      }
    },
    "$edgeHub": {
      "properties.desired": {
        "schemaVersion": "1.0",
        "routes": {
          "LVAToHub": "FROM /messages/modules/lvaEdge/outputs/* INTO $upstream"
        },
        "storeAndForwardConfiguration": {
          "timeToLiveSecs": 7200
        }
      }
    }
  }
}

Note the IpcMode set in lvaEdge and rtpsim. The actual containers used do not matter; any containers that use that use these ipcmode settings will trigger the issue.

Steps to reproduce

iotedgehubdev start -d config/minimal.json 

Network azure-iot-edge-dev is external, skipping
ERROR: Service 'rtspsim' uses the IPC namespace of container 'lvaEdge' which does not exist.
ERROR: Error while executing command: docker-compose -f /var/lib/iotedgehubdev/data/docker-compose.yml pull edgeHubDev. Command '['docker-compose', '-f', '/var/lib/iotedgehubdev/data/docker-compose.yml', 'pull', 'edgeHubDev']' returned non-zero exit status 1.

Possible solution

The IoT Edge deployment manifest module createOptions uses docker create options. The docker create container options allow the following values:

"none": own private IPC namespace, with /dev/shm not mounted
"private": own private IPC namespace
"shareable": own private IPC namespace, with a possibility to share it with other containers
"container:<name|id>": join another (shareable) container's IPC namespace
"host": use the host system's IPC namespace

However iotedgehubdev uses docker-compose to start the deployment. docker-compose requires the use of the setting ipc: "service:[service name]" when referring to another service in the same compose file. If the IpcMode value of rtpsim in the manifest above is changed from container:lvaEdge to service:lvaEdge iotedgehubdev is able to start up the solution as expected. This however will not work when deploying to a real edge device, as the service: syntax is not supported by docker create ("Error": "Invalid IPC mode: service:lvaEdge").

network _mode may also suffer from the same issue.