nextcloud/server

Can't open / sync big files on external storage (webdav)

Open

#30,282 建立於 2021年12月15日

在 GitHub 查看
 (24 留言) (8 反應) (0 負責人)PHP (4,865 fork)batch import
0. Needs triage28-feedbackbugfeature: external storagehelp wantedhotspot: file transfer performanceneeds review

倉庫指標

Star
 (34,953 star)
PR 合併指標
 (平均合併 17天 3小時) (30 天內合併 606 個 PR)

描述

How to use GitHub

  • Please use the 👍 reaction to show that you are affected by the same issue.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.

Steps to reproduce

  1. Add webdav external storage
  2. Tries to open big file there (e.g. 800 MB) or Sync folder via client
  3. See an error
  4. Check the logs and see:
GuzzleHttp\Exception\ConnectException: cURL error 28: Operation timed out after 30000 milliseconds with 379564427 out of 838860800 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://magentacloud.de/remote.php/webdav/System/NewTea.avi

Seems it is related to #30262 and #29969 with #26071 5. Use Workaround as per https://github.com/nextcloud/server/issues/26071#issuecomment-901785979 6. See that Web UI is now opening, but with a huge delay (whole file being downloaded), but client sync does not work still.

Expected behaviour

Do not download files (as per #30262) when folder simply opened, use cache instead. Do not download files when properties only needed (last modification date, or size)

Actual behaviour

External Storage could not be synchronized, even via Desktop Client it simply does not open and not synced any more. All files downloaded to /tmp or RAM and causing timeout.

Server configuration

Operating system: Ubuntu 20.04

Web server: Apache/2.4.51

Database: 10.3.32-MariaDB

PHP version: 7.4.26

Nextcloud version: 22.2.3.0

Updated from an older Nextcloud/ownCloud or fresh install: updated

Where did you install Nextcloud from: Official

Signing status:

No errors have been found.

List of activated apps:

Enabled:
  - accessibility: 1.8.0
  - activity: 2.15.0
  - admin_audit: 1.12.0
  - audioplayer: 3.2.2
  - bruteforcesettings: 2.2.0
  - calendar: 3.0.0
  - camerarawpreviews: 0.7.15
  - checksum: 1.1.2
  - cloud_federation_api: 1.5.0
  - comments: 1.12.0
  - contacts: 4.0.6
  - contactsinteraction: 1.3.0
  - cospend: 1.4.3
  - dashboard: 7.2.0
  - data_request: 1.9.0
  - dav: 1.19.0
  - deck: 1.5.5
  - drawio: 1.0.1
  - external: 3.9.0
  - extract: 1.3.2
  - facerecognition: 0.8.5
  - federatedfilesharing: 1.12.0
  - files: 1.17.0
  - files_automatedtagging: 1.12.0
  - files_downloadactivity: 1.11.1
  - files_external: 1.13.0
  - files_mindmap: 0.0.25
  - files_pdfviewer: 2.3.1
  - files_retention: 1.11.1
  - files_rightclick: 1.1.0
  - files_sharing: 1.14.0
  - files_trashbin: 1.12.0
  - files_versions: 1.15.0
  - files_videoplayer: 1.11.0
  - firstrunwizard: 2.11.0
  - forms: 2.4.0
  - gpxedit: 0.0.14
  - gpxpod: 4.3.0
  - integration_github: 1.0.2
  - integration_gitlab: 1.0.3
  - keeweb: 0.6.8
  - logreader: 2.7.0
  - lookup_server_connector: 1.10.0
  - mail: 1.11.0
  - maps: 0.1.9
  - nextcloud_announcements: 1.11.0
  - notes: 4.2.0
  - notifications: 2.10.1
  - oauth2: 1.10.0
  - ocdownloader: 1.7.12
  - password_policy: 1.12.0
  - phonetrack: 0.6.9
  - photos: 1.4.0
  - polls: 3.3.0
  - previewgenerator: 3.3.0
  - privacy: 1.6.0
  - provisioning_api: 1.12.0
  - recommendations: 1.1.0
  - serverinfo: 1.12.0
  - settings: 1.4.0
  - sharebymail: 1.12.0
  - side_menu: 2.0.1
  - spreed: 12.1.2
  - support: 1.5.0
  - survey_client: 1.10.0
  - systemtags: 1.12.0
  - text: 3.3.0
  - theming: 1.13.0
  - twofactor_backupcodes: 1.11.0
  - twofactor_totp: 6.1.0
  - unsplash: 1.2.4
  - updatenotification: 1.12.0
  - user_status: 1.2.0
  - video_converter: 1.0.3
  - viewer: 1.6.0
  - weather_status: 1.2.0
  - workflowengine: 2.4.0
Disabled:
  - circles
  - encryption
  - federation
  - files_accesscontrol
  - flowupload
  - gpxmotion
  - impersonate
  - integration_jira
  - radio
  - recognize
  - user_ldap
  - weather

Nextcloud configuration:

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "22.2.3.0",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "filesystem_check_changes": 0,
        "filelocking.enabled": true,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "timeout": 1.5
        },
        "default_phone_region": "DE",
        "mail_smtpmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "theme": "",
        "logfile": "\/var\/nextcloud\/data\/nextcloud.log",
        "loglevel": 1,
        "trashbin_retention_obligation": "14, auto",
        "versions_retention_obligation": "14, auto",
        "data-fingerprint": "7be51475e95ea13b254cd880319d98ec",
        "enable_previews": true,
        "enabledPreviewProviders": [
            "OC\\Preview\\PNG",
            "OC\\Preview\\JPEG",
            "OC\\Preview\\GIF",
            "OC\\Preview\\BMP",
            "OC\\Preview\\XBitmap",
            "OC\\Preview\\Movie",
            "OC\\Preview\\PDF",
            "OC\\Preview\\MP3",
            "OC\\Preview\\TXT",
            "OC\\Preview\\MarkDown"
        ],
        "preview_max_x": 1920,
        "preview_max_y": 1080,
        "jpeg_quality": 90,
        "auth.bruteforce.protection.enabled": true,
        "simpleSignUpLink.shown": false,
        "mail_smtpsecure": "tls",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "twofactor_enforced": true,
        "twofactor_enforced_groups": [
            "admin"
        ],
        "twofactor_enforced_excluded_groups": [],
        "has_rebuilt_cache": true,
        "updater.release.channel": "stable",
        "mail_sendmailmode": "smtp"
    }
}

Are you using external storage, if yes which one: local/webdav

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

Logs

Nextcloud log (data/nextcloud.log)

{"reqId":"3bdICeP14G6vPYqfCKqQ","level":3,"time":"2021-12-15T09:30:33+00:00","remoteAddr":"","user":"--","app":"files","method":"","url":"--","message":"cURL error 28: Operation timed out after 30000 milliseconds with 379564427 out of 838860800 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://magentacloud.de/remote.php/webdav/System/NewTea.avi","userAgent":"--","version":"22.2.3.0","exception":{"Exception":"GuzzleHttp\\Exception\\ConnectException","Message":"cURL error 28: Operation timed out after 30000 milliseconds with 379564427 out of 838860800 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://magentacloud.de/remote.php/webdav/System/NewTea.avi","Code":0,"Trace":[{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php","line":158,"function":"createRejection","class":"GuzzleHttp\\Handler\\CurlFactory","type":"::"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php","line":110,"function":"finishError","class":"GuzzleHttp\\Handler\\CurlFactory","type":"::"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlHandler.php","line":47,"function":"finish","class":"GuzzleHttp\\Handler\\CurlFactory","type":"::"},{"file":"/var/www/nextcloud/lib/private/Http/Client/DnsPinMiddleware.php","line":146,"function":"__invoke","class":"GuzzleHttp\\Handler\\CurlHandler","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php","line":35,"function":"OC\\Http\\Client\\{closure}","class":"OC\\Http\\Client\\DnsPinMiddleware","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php","line":31,"function":"__invoke","class":"GuzzleHttp\\PrepareBodyMiddleware","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/RedirectMiddleware.php","line":71,"function":"GuzzleHttp\\{closure}","class":"GuzzleHttp\\Middleware","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php","line":63,"function":"__invoke","class":"GuzzleHttp\\RedirectMiddleware","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/HandlerStack.php","line":75,"function":"GuzzleHttp\\{closure}","class":"GuzzleHttp\\Middleware","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php","line":331,"function":"__invoke","class":"GuzzleHttp\\HandlerStack","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php","line":168,"function":"transfer","class":"GuzzleHttp\\Client","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php","line":187,"function":"requestAsync","class":"GuzzleHttp\\Client","type":"->"},{"file":"/var/www/nextcloud/lib/private/Http/Client/Client.php","line":223,"function":"request","class":"GuzzleHttp\\Client","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/DAV.php","line":356,"function":"get","class":"OC\\Http\\Client\\Client","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/LocalTempFileTrait.php","line":65,"function":"fopen","class":"OC\\Files\\Storage\\DAV","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/LocalTempFileTrait.php","line":48,"function":"toTmpFile","class":"OC\\Files\\Storage\\Common","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Common.php","line":269,"function":"getCachedFile","class":"OC\\Files\\Storage\\Common","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":367,"function":"getLocalFile","class":"OC\\Files\\Storage\\Common","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Availability.php","line":374,"function":"getLocalFile","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":367,"function":"getLocalFile","class":"OC\\Files\\Storage\\Wrapper\\Availability","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Check/FileMimeType.php","line":131,"function":"getLocalFile","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Check/FileMimeType.php","line":109,"function":"getActualValue","class":"OCA\\WorkflowEngine\\Check\\FileMimeType","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Service/RuleMatcher.php","line":245,"function":"executeCheck","class":"OCA\\WorkflowEngine\\Check\\FileMimeType","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Service/RuleMatcher.php","line":180,"function":"check","class":"OCA\\WorkflowEngine\\Service\\RuleMatcher","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Service/RuleMatcher.php","line":128,"function":"getMatchingOperations","class":"OCA\\WorkflowEngine\\Service\\RuleMatcher","type":"->"},{"file":"/var/www/nextcloud/apps/files_automatedtagging/lib/Operation.php","line":95,"function":"getFlows","class":"OCA\\WorkflowEngine\\Service\\RuleMatcher","type":"->"},{"file":"/var/www/nextcloud/apps/files_automatedtagging/lib/Listener/CacheListener.php","line":43,"function":"checkOperations","class":"OCA\\FilesAutomatedTagging\\Operation","type":"->"},{"file":"/var/www/nextcloud/lib/private/EventDispatcher/ServiceEventListener.php","line":75,"function":"handle","class":"OCA\\FilesAutomatedTagging\\Listener\\CacheListener","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php","line":251,"function":"__invoke","class":"OC\\EventDispatcher\\ServiceEventListener","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php","line":73,"function":"callListeners","class":"Symfony\\Component\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php","line":88,"function":"dispatch","class":"Symfony\\Component\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php","line":100,"function":"dispatch","class":"OC\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Cache.php","line":324,"function":"dispatchTyped","class":"OC\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":294,"function":"insert","class":"OC\\Files\\Cache\\Cache","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":224,"function":"addToCache","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":427,"function":"scanFile","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":388,"function":"handleChildren","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":340,"function":"scanChildren","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":510,"function":"scan","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":521,"function":"OC\\Files\\Cache\\{closure}","class":"OC\\Files\\Cache\\Scanner","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":509,"function":"runBackgroundScanJob","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Utils/Scanner.php","line":185,"function":"backgroundScan","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/apps/files/lib/BackgroundJob/ScanFiles.php","line":89,"function":"backgroundScan","class":"OC\\Files\\Utils\\Scanner","type":"->"},{"file":"/var/www/nextcloud/apps/files/lib/BackgroundJob/ScanFiles.php","line":125,"function":"runScanner","class":"OCA\\Files\\BackgroundJob\\ScanFiles","type":"->"},{"file":"/var/www/nextcloud/lib/private/BackgroundJob/Job.php","line":51,"function":"run","class":"OCA\\Files\\BackgroundJob\\ScanFiles","type":"->"},{"file":"/var/www/nextcloud/lib/private/BackgroundJob/TimedJob.php","line":58,"function":"execute","class":"OC\\BackgroundJob\\Job","type":"->"},{"file":"/var/www/nextcloud/cron.php","line":127,"function":"execute","class":"OC\\BackgroundJob\\TimedJob","type":"->"}],"File":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php","Line":210,"CustomMessage":"--"},"id":"61b9f1e0b44fa"}

Nextcloud Desktop Client log

||MagentaCloud|2|2|1638616226|_invalid_|0|10006743xxxxxxxxxxx|4||0|0|0||
#=#=#=# Syncrun finished 2021-12-15T13:59:38Z (last step: 157 msec, total: 2134 msec)
#=#=#=# Syncrun started 2021-12-15T13:59:40Z
#=#=#=#=# Propagation starts 2021-12-15T14:00:24Z (last step: 43733 msec, total: 43733 msec)
#=#=#=# Syncrun finished 2021-12-15T14:00:24Z (last step: 24 msec, total: 43757 msec)
#=#=#=# Syncrun started 2021-12-15T14:01:03Z
#=#=#=#=# Propagation starts 2021-12-15T14:01:03Z (last step: 371 msec, total: 371 msec)
||MagentaCloud|8|2|1638616226|61ab4ca2b0b9c|0|10006743xxxxxxxxxxx|4||0|0|0||
#=#=#=# Syncrun finished 2021-12-15T14:01:03Z (last step: 28 msec, total: 399 msec)

貢獻者指南

研究方向
調查從 WebDAV 外部儲存擷取大檔案時 cURL 逾時的根本原因。查閱相關議題 #30262、#29969、#26071 及其關聯的 PR。檢查 apps/files external/lib/Storage/WebDAV.php 中的檔案處理邏輯以及 lib/private/Files/Storage/Common.php 中的快取機制。研究增加 cURL 逾時時間或實作分塊下載。考慮新增逾時限制的設定選項，並改進逾時發生時的後備行為。
技術棧
php
領域
backendperformance
議題類型
錯誤
難度面向新貢獻者的預計實作難度，1 表示很小改動，5 表示專家級工作。
4
預計時間有經驗貢獻者完成調查、實作、測試並準備 pull request 的粗略時間範圍。
超過 1 週
活動狀態議題目前的可參與程度：新鮮、活躍、陳舊、阻塞或等待維護者輸入。
較久未更新
清晰度議題是否清楚說明預期改動、驗收標準和下一步。
大致清晰
前置要求
PHPWebDAV protocolNextcloud files external appHTTP timeout handling
新手友善度1-100 的估計分數，表示該議題對首次貢獻者的友善程度。
20