nextcloud/server

Can't open / sync big files on external storage (webdav)

Open

#30,282 创建于 2021年12月15日

在 GitHub 查看
 (24 评论) (8 反应) (0 负责人)PHP (4,865 fork)batch import
0. Needs triage28-feedbackbugfeature: external storagehelp wantedhotspot: file transfer performanceneeds review

仓库指标

Star
 (34,953 star)
PR 合并指标
 (平均合并 17天 3小时) (30 天内合并 606 个 PR)

描述

How to use GitHub

  • Please use the 👍 reaction to show that you are affected by the same issue.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.

Steps to reproduce

  1. Add webdav external storage
  2. Tries to open big file there (e.g. 800 MB) or Sync folder via client
  3. See an error
  4. Check the logs and see:
GuzzleHttp\Exception\ConnectException: cURL error 28: Operation timed out after 30000 milliseconds with 379564427 out of 838860800 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://magentacloud.de/remote.php/webdav/System/NewTea.avi

Seems it is related to #30262 and #29969 with #26071 5. Use Workaround as per https://github.com/nextcloud/server/issues/26071#issuecomment-901785979 6. See that Web UI is now opening, but with a huge delay (whole file being downloaded), but client sync does not work still.

Expected behaviour

Do not download files (as per #30262) when folder simply opened, use cache instead. Do not download files when properties only needed (last modification date, or size)

Actual behaviour

External Storage could not be synchronized, even via Desktop Client it simply does not open and not synced any more. All files downloaded to /tmp or RAM and causing timeout.

Server configuration

Operating system: Ubuntu 20.04

Web server: Apache/2.4.51

Database: 10.3.32-MariaDB

PHP version: 7.4.26

Nextcloud version: 22.2.3.0

Updated from an older Nextcloud/ownCloud or fresh install: updated

Where did you install Nextcloud from: Official

Signing status:

No errors have been found.

List of activated apps:

Enabled:
  - accessibility: 1.8.0
  - activity: 2.15.0
  - admin_audit: 1.12.0
  - audioplayer: 3.2.2
  - bruteforcesettings: 2.2.0
  - calendar: 3.0.0
  - camerarawpreviews: 0.7.15
  - checksum: 1.1.2
  - cloud_federation_api: 1.5.0
  - comments: 1.12.0
  - contacts: 4.0.6
  - contactsinteraction: 1.3.0
  - cospend: 1.4.3
  - dashboard: 7.2.0
  - data_request: 1.9.0
  - dav: 1.19.0
  - deck: 1.5.5
  - drawio: 1.0.1
  - external: 3.9.0
  - extract: 1.3.2
  - facerecognition: 0.8.5
  - federatedfilesharing: 1.12.0
  - files: 1.17.0
  - files_automatedtagging: 1.12.0
  - files_downloadactivity: 1.11.1
  - files_external: 1.13.0
  - files_mindmap: 0.0.25
  - files_pdfviewer: 2.3.1
  - files_retention: 1.11.1
  - files_rightclick: 1.1.0
  - files_sharing: 1.14.0
  - files_trashbin: 1.12.0
  - files_versions: 1.15.0
  - files_videoplayer: 1.11.0
  - firstrunwizard: 2.11.0
  - forms: 2.4.0
  - gpxedit: 0.0.14
  - gpxpod: 4.3.0
  - integration_github: 1.0.2
  - integration_gitlab: 1.0.3
  - keeweb: 0.6.8
  - logreader: 2.7.0
  - lookup_server_connector: 1.10.0
  - mail: 1.11.0
  - maps: 0.1.9
  - nextcloud_announcements: 1.11.0
  - notes: 4.2.0
  - notifications: 2.10.1
  - oauth2: 1.10.0
  - ocdownloader: 1.7.12
  - password_policy: 1.12.0
  - phonetrack: 0.6.9
  - photos: 1.4.0
  - polls: 3.3.0
  - previewgenerator: 3.3.0
  - privacy: 1.6.0
  - provisioning_api: 1.12.0
  - recommendations: 1.1.0
  - serverinfo: 1.12.0
  - settings: 1.4.0
  - sharebymail: 1.12.0
  - side_menu: 2.0.1
  - spreed: 12.1.2
  - support: 1.5.0
  - survey_client: 1.10.0
  - systemtags: 1.12.0
  - text: 3.3.0
  - theming: 1.13.0
  - twofactor_backupcodes: 1.11.0
  - twofactor_totp: 6.1.0
  - unsplash: 1.2.4
  - updatenotification: 1.12.0
  - user_status: 1.2.0
  - video_converter: 1.0.3
  - viewer: 1.6.0
  - weather_status: 1.2.0
  - workflowengine: 2.4.0
Disabled:
  - circles
  - encryption
  - federation
  - files_accesscontrol
  - flowupload
  - gpxmotion
  - impersonate
  - integration_jira
  - radio
  - recognize
  - user_ldap
  - weather

Nextcloud configuration:

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "22.2.3.0",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "filesystem_check_changes": 0,
        "filelocking.enabled": true,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "timeout": 1.5
        },
        "default_phone_region": "DE",
        "mail_smtpmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "theme": "",
        "logfile": "\/var\/nextcloud\/data\/nextcloud.log",
        "loglevel": 1,
        "trashbin_retention_obligation": "14, auto",
        "versions_retention_obligation": "14, auto",
        "data-fingerprint": "7be51475e95ea13b254cd880319d98ec",
        "enable_previews": true,
        "enabledPreviewProviders": [
            "OC\\Preview\\PNG",
            "OC\\Preview\\JPEG",
            "OC\\Preview\\GIF",
            "OC\\Preview\\BMP",
            "OC\\Preview\\XBitmap",
            "OC\\Preview\\Movie",
            "OC\\Preview\\PDF",
            "OC\\Preview\\MP3",
            "OC\\Preview\\TXT",
            "OC\\Preview\\MarkDown"
        ],
        "preview_max_x": 1920,
        "preview_max_y": 1080,
        "jpeg_quality": 90,
        "auth.bruteforce.protection.enabled": true,
        "simpleSignUpLink.shown": false,
        "mail_smtpsecure": "tls",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "twofactor_enforced": true,
        "twofactor_enforced_groups": [
            "admin"
        ],
        "twofactor_enforced_excluded_groups": [],
        "has_rebuilt_cache": true,
        "updater.release.channel": "stable",
        "mail_sendmailmode": "smtp"
    }
}

Are you using external storage, if yes which one: local/webdav

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

Logs

Nextcloud log (data/nextcloud.log)

{"reqId":"3bdICeP14G6vPYqfCKqQ","level":3,"time":"2021-12-15T09:30:33+00:00","remoteAddr":"","user":"--","app":"files","method":"","url":"--","message":"cURL error 28: Operation timed out after 30000 milliseconds with 379564427 out of 838860800 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://magentacloud.de/remote.php/webdav/System/NewTea.avi","userAgent":"--","version":"22.2.3.0","exception":{"Exception":"GuzzleHttp\\Exception\\ConnectException","Message":"cURL error 28: Operation timed out after 30000 milliseconds with 379564427 out of 838860800 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://magentacloud.de/remote.php/webdav/System/NewTea.avi","Code":0,"Trace":[{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php","line":158,"function":"createRejection","class":"GuzzleHttp\\Handler\\CurlFactory","type":"::"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php","line":110,"function":"finishError","class":"GuzzleHttp\\Handler\\CurlFactory","type":"::"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlHandler.php","line":47,"function":"finish","class":"GuzzleHttp\\Handler\\CurlFactory","type":"::"},{"file":"/var/www/nextcloud/lib/private/Http/Client/DnsPinMiddleware.php","line":146,"function":"__invoke","class":"GuzzleHttp\\Handler\\CurlHandler","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php","line":35,"function":"OC\\Http\\Client\\{closure}","class":"OC\\Http\\Client\\DnsPinMiddleware","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php","line":31,"function":"__invoke","class":"GuzzleHttp\\PrepareBodyMiddleware","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/RedirectMiddleware.php","line":71,"function":"GuzzleHttp\\{closure}","class":"GuzzleHttp\\Middleware","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php","line":63,"function":"__invoke","class":"GuzzleHttp\\RedirectMiddleware","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/HandlerStack.php","line":75,"function":"GuzzleHttp\\{closure}","class":"GuzzleHttp\\Middleware","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php","line":331,"function":"__invoke","class":"GuzzleHttp\\HandlerStack","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php","line":168,"function":"transfer","class":"GuzzleHttp\\Client","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php","line":187,"function":"requestAsync","class":"GuzzleHttp\\Client","type":"->"},{"file":"/var/www/nextcloud/lib/private/Http/Client/Client.php","line":223,"function":"request","class":"GuzzleHttp\\Client","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/DAV.php","line":356,"function":"get","class":"OC\\Http\\Client\\Client","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/LocalTempFileTrait.php","line":65,"function":"fopen","class":"OC\\Files\\Storage\\DAV","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/LocalTempFileTrait.php","line":48,"function":"toTmpFile","class":"OC\\Files\\Storage\\Common","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Common.php","line":269,"function":"getCachedFile","class":"OC\\Files\\Storage\\Common","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":367,"function":"getLocalFile","class":"OC\\Files\\Storage\\Common","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Availability.php","line":374,"function":"getLocalFile","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":367,"function":"getLocalFile","class":"OC\\Files\\Storage\\Wrapper\\Availability","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Check/FileMimeType.php","line":131,"function":"getLocalFile","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Check/FileMimeType.php","line":109,"function":"getActualValue","class":"OCA\\WorkflowEngine\\Check\\FileMimeType","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Service/RuleMatcher.php","line":245,"function":"executeCheck","class":"OCA\\WorkflowEngine\\Check\\FileMimeType","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Service/RuleMatcher.php","line":180,"function":"check","class":"OCA\\WorkflowEngine\\Service\\RuleMatcher","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Service/RuleMatcher.php","line":128,"function":"getMatchingOperations","class":"OCA\\WorkflowEngine\\Service\\RuleMatcher","type":"->"},{"file":"/var/www/nextcloud/apps/files_automatedtagging/lib/Operation.php","line":95,"function":"getFlows","class":"OCA\\WorkflowEngine\\Service\\RuleMatcher","type":"->"},{"file":"/var/www/nextcloud/apps/files_automatedtagging/lib/Listener/CacheListener.php","line":43,"function":"checkOperations","class":"OCA\\FilesAutomatedTagging\\Operation","type":"->"},{"file":"/var/www/nextcloud/lib/private/EventDispatcher/ServiceEventListener.php","line":75,"function":"handle","class":"OCA\\FilesAutomatedTagging\\Listener\\CacheListener","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php","line":251,"function":"__invoke","class":"OC\\EventDispatcher\\ServiceEventListener","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php","line":73,"function":"callListeners","class":"Symfony\\Component\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php","line":88,"function":"dispatch","class":"Symfony\\Component\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php","line":100,"function":"dispatch","class":"OC\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Cache.php","line":324,"function":"dispatchTyped","class":"OC\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":294,"function":"insert","class":"OC\\Files\\Cache\\Cache","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":224,"function":"addToCache","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":427,"function":"scanFile","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":388,"function":"handleChildren","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":340,"function":"scanChildren","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":510,"function":"scan","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":521,"function":"OC\\Files\\Cache\\{closure}","class":"OC\\Files\\Cache\\Scanner","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":509,"function":"runBackgroundScanJob","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Utils/Scanner.php","line":185,"function":"backgroundScan","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/apps/files/lib/BackgroundJob/ScanFiles.php","line":89,"function":"backgroundScan","class":"OC\\Files\\Utils\\Scanner","type":"->"},{"file":"/var/www/nextcloud/apps/files/lib/BackgroundJob/ScanFiles.php","line":125,"function":"runScanner","class":"OCA\\Files\\BackgroundJob\\ScanFiles","type":"->"},{"file":"/var/www/nextcloud/lib/private/BackgroundJob/Job.php","line":51,"function":"run","class":"OCA\\Files\\BackgroundJob\\ScanFiles","type":"->"},{"file":"/var/www/nextcloud/lib/private/BackgroundJob/TimedJob.php","line":58,"function":"execute","class":"OC\\BackgroundJob\\Job","type":"->"},{"file":"/var/www/nextcloud/cron.php","line":127,"function":"execute","class":"OC\\BackgroundJob\\TimedJob","type":"->"}],"File":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php","Line":210,"CustomMessage":"--"},"id":"61b9f1e0b44fa"}

Nextcloud Desktop Client log

||MagentaCloud|2|2|1638616226|_invalid_|0|10006743xxxxxxxxxxx|4||0|0|0||
#=#=#=# Syncrun finished 2021-12-15T13:59:38Z (last step: 157 msec, total: 2134 msec)
#=#=#=# Syncrun started 2021-12-15T13:59:40Z
#=#=#=#=# Propagation starts 2021-12-15T14:00:24Z (last step: 43733 msec, total: 43733 msec)
#=#=#=# Syncrun finished 2021-12-15T14:00:24Z (last step: 24 msec, total: 43757 msec)
#=#=#=# Syncrun started 2021-12-15T14:01:03Z
#=#=#=#=# Propagation starts 2021-12-15T14:01:03Z (last step: 371 msec, total: 371 msec)
||MagentaCloud|8|2|1638616226|61ab4ca2b0b9c|0|10006743xxxxxxxxxxx|4||0|0|0||
#=#=#=# Syncrun finished 2021-12-15T14:01:03Z (last step: 28 msec, total: 399 msec)

贡献者指南

研究方向
调查从 WebDAV 外部存储获取大文件时 cURL 超时的根本原因。查阅相关议题 #30262、#29969、#26071 及其关联的 PR。检查 apps/files external/lib/Storage/WebDAV.php 中的文件处理逻辑以及 lib/private/Files/Storage/Common.php 中的缓存机制。研究增加 cURL 超时时间或实现分块下载。考虑添加超时限制的配置选项，并改进超时发生时的回退行为。
技术栈
php
领域
backendperformance
议题类型
缺陷
难度面向新贡献者的预计实现难度，1 表示很小改动，5 表示专家级工作。
4
预计时间有经验贡献者完成调查、实现、测试并准备 pull request 的粗略时间范围。
超过 1 周
活动状态议题当前的可参与程度：新鲜、活跃、陈旧、阻塞或等待维护者输入。
较久未更新
清晰度议题是否清楚说明期望改动、验收标准和下一步。
基本清晰
前置要求
PHPWebDAV protocolNextcloud files external appHTTP timeout handling
新手友好度1-100 的估计分数，表示该议题对首次贡献者的友好程度。
20