Very slow upgrade process

Bắt đầu với các init log, cấu hình compose/stack của Docker Swarm và cấu hình máy chủ, tập trung vào khoảng trống 25 phút trước khi tìm kiếm pre-upgrade hook. Tái hiện quá trình nâng cấp với các image và giới hạn tài nguyên đã được báo cáo, sau đó ghi lại nguyên nhân và một bước cấu hình hoặc chẩn đoán có thể thực hiện; issue không đề cập đến việc kiểm thử.

Issue

When trying to upgrade the nextcloud version, i saw that the upgrade was stuck for ~30m before continuing to process. (it was already happening before but only stuck for 5m)

Currently, i have no idea what is causing this issue, so if someone already have the same issue, i could try to change config, and else, if someone could provide some commands i could run to see what could cause the issue on a next upgrade, that would be nice.

Files / Logs

Here is the init logs : (check the time diff between line 6 and 7)

2025-06-15T11:36:13.887214965Z Conf remoteip disabled.
2025-06-15T11:36:13.887277700Z To activate the new configuration, you need to run:
2025-06-15T11:36:13.887292103Z   service apache2 reload
2025-06-15T11:36:13.897353629Z Configuring Redis as session handler
2025-06-15T11:36:14.313096226Z Initializing nextcloud 31.0.6.2 ...
2025-06-15T11:36:14.313114123Z Upgrading nextcloud from 31.0.5.1 ...
2025-06-15T12:01:09.655694259Z => Searching for hook scripts (*.sh) to run, located in the folder "/docker-entrypoint-hooks.d/pre-upgrade"
2025-06-15T12:01:09.659895462Z ==> Skipped: the "pre-upgrade" folder is empty (or does not exist)
2025-06-15T12:01:33.358814697Z Nextcloud or one of the apps require upgrade - only a limited number of commands are available
2025-06-15T12:01:33.358851309Z You may use your browser or the occ upgrade command to do the upgrade
2025-06-15T12:01:33.767487087Z Setting log level to debug
2025-06-15T12:01:34.520048300Z Turned on maintenance mode
2025-06-15T12:01:38.139143538Z Updating database schema
2025-06-15T12:01:38.247622306Z Updated database
2025-06-15T12:01:55.975508497Z Starting code integrity check...
2025-06-15T12:04:32.869696894Z Finished code integrity check
2025-06-15T12:04:33.119839658Z Update successful
2025-06-15T12:04:33.261717198Z Turned off maintenance mode
2025-06-15T12:04:33.262340213Z Resetting log level
2025-06-15T12:04:35.705705010Z The following apps have been disabled:
2025-06-15T12:04:35.707650208Z => Searching for hook scripts (*.sh) to run, located in the folder "/docker-entrypoint-hooks.d/post-upgrade"
2025-06-15T12:04:35.708894322Z ==> Skipped: the "post-upgrade" folder is empty (or does not exist)
2025-06-15T12:04:35.708922769Z Initializing finished

compose/stack (i'm using docker swarm) :

services:
  db:
    image: mariadb:10.6
    restart: always
    command: --transaction-isolation=READ-COMMITTED --log-bin=binlog --binlog-format=ROW
    volumes:
      - db1:/var/lib/mysql
    environment:
      - MARIADB_RANDOM_ROOT_PASSWORD=1
      - MYSQL_PASSWORD=xx
      - MYSQL_DATABASE=xx
      - MYSQL_USER=xx
    deploy:
      replicas: 1
      placement:
        constraints: [node.labels.worker == true]
      resources:
        limits:
          cpus: '0.50'
          memory: 250M
        reservations:
          memory: 100M

  redis:
    image: redis
    restart: always
    volumes:
      - redis1:/var/lib/redis/data
    command: redis-server --requirepass xx
    deploy:
      replicas: 1
      placement:
        constraints: [node.labels.worker == true]
      resources:
        limits:
          cpus: '0.50'
          memory: 250M
        reservations:
          memory: 100M

  app:
    image: nextcloud
    restart: always
    networks:
      - default
      - traefik
      - ldap
    links:
      - db
      - redis
    volumes:
      - nextcloud1:/var/www/html
    environment:
      - MYSQL_PASSWORD=xx
      - MYSQL_DATABASE=xx
      - MYSQL_USER=xx
      - MYSQL_HOST=db
      - REDIS_HOST=redis
      - REDIS_HOST_PASSWORD=xx
      - APACHE_DISABLE_REWRITE_IP=1
      - PHP_MEMORY_LIMIT=2048M
    deploy:
      replicas: 1
      placement:
        constraints: [node.labels.worker == true]
      labels:
        - traefik.enable=true
        - .... others traefik labels
      
  cron:
    image: nextcloud
    restart: always
    networks:
      - default
      - ldap
    links:
      - db
      - redis
    volumes:
      - nextcloud1:/var/www/html
    entrypoint: /cron.sh
    environment:
      - MYSQL_PASSWORD=xx
      - MYSQL_DATABASE=xx
      - MYSQL_USER=xx
      - MYSQL_HOST=db
      - REDIS_HOST=redis
      - REDIS_HOST_PASSWORD=xx
    deploy:
      replicas: 1
      placement:
        constraints: [node.labels.worker == true]

networks:
  default:
  traefik:
    external: true
  ldap:
    external: true

# omit volumes, not really relevent as they are connected and working

server config :

{
    "system": {
        "default_phone_region": "FR",
        "htaccess.RewriteBase": "\/",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "upgrade.disable-web": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "cloud.<domain>",
            "cloud.<server1>.<domain>",
            "cloud.<server2>.<domain>",
            "cloud.<server3>.<domain>"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "31.0.6.2",
        "overwrite.cli.url": "https:\/\/cloud.<domain>",
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "forwarded_for_headers": [
            "HTTP_X_FORWARDED_FOR"
        ],
        "overwriteprotocol": "https",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory",
        "defaultapp": "files,calendar",
        "maintenance": false,
        "filelocking.enabled": true,
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_sendmailmode": "smtp",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpsecure": "ssl",
        "mail_smtpport": "465",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "maintenance_window_start": 2,
        "app_install_overwrite": [
            "google_synchronization"
        ],
        "loglevel": 1
    }
}