Unify builtin network customization, improve daemon.json structure

Description

Related to:

• https://github.com/moby/moby/pull/50359#discussion_r2225781675
• https://github.com/moby/moby/issues/50716
• https://github.com/moby/moby/pull/50790

Problem statement

There are currently a few different mechanisms to customize predefined and custom networks, each targeting a specific set of networks.

1. Daemon options

Daemon flags, or top-level daemon.json settings, but these will only apply to the default network, and not all options are available (e.g. com.docker.network.windowsshim.disable_gatewaydns for Windows networks).

These default network options are mixed up with netdriver options, IPAM options, etc... But there's no proper hierarchization between the various categories, so it adds needless confusion for end-users and AIs (e.g. Gordon is getting confused), as it's hard to grok the scope of each option.

Daemon options only applied to the default network (see here):

• bridge
• fixed-cidr
• ipv6
• fixed-cidr-v6
• mtu
• bip
• bip6
• default-gateway
• default-gateway-v6
• icc

Daemon options applied to the bridge driver (see here):

• iptables
• ip6tables
• ip-forward
• ip-forward-no-drop
• ip-masq
• userland-proxy
• userland-proxy-path
• allow-direct-routing
• bridge-accept-fwmark

Other daemon options (see here):

• default-address-pools
• network-control-plane-mtu (although this setting is part of NetworkConfig, it's related to Swarm and should not be mixed with networking settings)
• default-network-opts
• firewall-backend

Network options which have no equivalent daemon option:

• com.docker.network.advertise_addr_nmsgs
• com.docker.network.advertise_addr_ms
• com.docker.network.host_ipv4
• com.docker.network.host_ipv6
• com.docker.network.bridge.gateway_mode_ipv4
• com.docker.network.bridge.gateway_mode_ipv6
• com.docker.network.bridge.inhibit_ipv4
• com.docker.network.bridge.trusted_host_interfaces
• Most of Windows netlabels

2. default-network-opts

default-network-opts can be used to set default network options, but only on custom networks. This may be surprising from a user perspective (see https://github.com/moby/moby/issues/50716) as the name doesn't reflect that. Moreover, it only allows users to set 'netlabel' keys, and provide no way to move from that model in the future, or to add non netlabel-based options.

3. ingress and docker_gwbridge networks

Each of these networks have their own customization procedure.

For ingress networks, removing the existing network, and re-creating it with the desired options using docker network create is enough. See here.

However, customizing docker_gwbridge is a bit more complex... It involves stopping the Engine, deleting the bridge interface, and then restarting the Engine while making sure that no services will be scheduled on that node to ensure that docker_gwbridge won't be recreated automatically. See here.

Proposal

Moving networking settings under a top-level networking section

In order to better organize networking settings, I'm proposing that we move all of them under a top-level networking section.

default-network-opts

In order to make default-network-opts less confusing, I'm proposing that it's applied to all networks, including builtin networks (i.e. default bridge network, and docker_gwbridge).

Following previous proposal, this setting would be renamed / moved to networking.default-opts.

Also, in order to not paint us into a corner once again, I'm proposing that it takes a map of github.com/moby/moby/api/types/network.CreateOptions keyed by netdriver names, or at least a map of map[string]map[string]any keyed by netdriver names too, but whose 2nd-level map would only accept the options key (giving us the ability to add more keys later).

{
    "networking": {
        "default-opts": {
            "bridge": {
                "enable_ipv4": true,
                "enable_ipv6": true,
                "options": {
                    "com.docker.network.advertise_addr_ms": 100
                },
            }
        }
    }
}

Predefined networks

In order to improve the distinction between settings for the default bridge network and settings for the bridge netdriver, and to improve the UX for ingress and docker_gwbridge customization, I'm proposing that we add a new networking.predefined-nws setting that would take a map of network definitions. The Engine would ensure these networks always exist, i.e. it'd create them on startup, and block their deletion. This would provide a natural way to customize builtin networks, as well as define ready-to-use custom networks. These definitions would also use github.com/moby/moby/api/types/network.CreateOptions, and default-opts would be applied. Thus, builtin networks would be created by merging multiple sources following the following order of precedence (from lowest to highest):

1. hardcoded defaults
2. networking.default-opts
3. networking.predefined-nws

Both docker_gwbridge and ingress networks would be special-cased to not create them on startup, but when (re-)joining a Swarm cluster (so either after startup when joining a cluster, or when the node re-joins a cluster).

{
    "networking": {
        "predefined-nws": {
            "docker_gwbridge": {
                "driver": "bridge",
                "options": {
                    "com.docker.network.bridge.enable_icc": false
                }
            }
        }
    }
}

Re-organize driver settings

In order to clarify the scope of each settings, I'm proposing that driver-level settings are moved to a new networking.drivers section that would take a map of driver names, and driver options.

However, I think there's a prerequisite to this proposal: we should move from the horrible map[string]any pattern, to driver-specific structs, similarly to what I did for portmappers (see here). Either the daemon pkg, or libnetwork pkg would be responsible for instantiating these structs. In the former case, these structs could be transitively stored in a map[string]any, to let libnetwork type-assert them.

Then, the driver options declared in networking.drivers would be based on each individual driver struct.

{
    "networking": {
        "drivers": {
            "bridge": {
                "allow-direct-routing": true,
            }
        }
    }
}

Migration strategy

Top-level settings would be deprecated for at least one major version. During deprecation period, the Engine would automatically migrate the settings to the new networking section.

daemon.json options -> dockerd flags mapping

All dockerd flags would follow the same hierarchy as daemon.json, e.g. networking.drivers.bridge.allow-direct-routing=true, networking.predefined-nws.docker_gwbridge.mtu, etc.