Auto upload: ideas for predictable behaviour, duplicate-free re-scans and recovery

オープン
#17,801 コメント 1 件 リアクション 3 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
25/100
issue の種類
機能追加
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
android, kotlin
領域
backend, mobile

調査の方向性

This issue is a six increment proposal and does not name app files or tests. Start by reading the related issues and pull requests listed under the increments, then identify one increment with maintainers. Done means implementing that increment's stated behavior and verifying it with the relevant tests; the proposal itself needs discussion before the full design is settled.

索引モデルが issue の本文から書いたものです。

説明

0. Needs triage enhancement
How to use GitHub
  • Please use the 👍 reaction to show that you are interested into the same feature.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.

Is your feature request related to a problem? Please describe.

Auto upload with "keep original" works most of the time. When it fails, it's hard to notice and hard to recover from:

  • A phone silently stops uploading, and nobody notices for weeks (#14945, #16334, #17212).
  • The only way to catch up is "Also upload existing files". That re-uploads every photo on the phone that the app has no record of, including photos already uploaded and later moved or deleted on the server (#16348, #6911).
  • Identical files become conflicts, or IMG (2).jpg copies with "Rename" (#4255, #8948, #12861).
  • Re-adding a folder, changing its target, or reinstalling loses the app's record of what it uploaded (#12967, #16802, #11090).

Reliability has improved a lot over the last year: the new auto upload worker, the rescan worker in 35.0.0, the stuck-worker fix coming in 35.0.2, and the work coordinated in #16550. Thank you for that. Hopefully silent stops will become rare. Uploads can still fail for reasons outside the app's control (OEM battery management, permissions, a full server, expired credentials), so it remains important that failures are visible and that recovery is predictable and doesn't re-upload everything. That's the focus of this proposal; it doesn't replace the ongoing bug fixing.

Many issues and PRs touch parts of this. This issue tries to connect them under one model of how I believe auto upload should behave, so each part can be discussed and built separately. It follows the increment format of #285.

My ideas depend on a few new concepts, such as a "window" deciding which files to consider for upload and an improved upload-log. These are explained in more detail below.

Expected behaviour per case I have tried to compile a list of different "use cases" and how I believe the app should behave in each case.
Case Behaviour
A folder is added Everything in the window (see below) is uploaded except files in the log or identical on the server
A new photo is taken Uploaded
A removed folder is added back Nothing already uploaded is sent again; the log is per file, not per rule
An old photo is moved or restored into a folder Judged by capture date and window, then the normal decision
Two rules cover the same file Uploaded once
A photo is edited on the phone after upload New content, so the collision policy applies
New phone or reinstall The log is restored from its server copy. Without one, moved or deleted files come back, and this is documented
Uploads stall completely A notification "Auto upload stalled since ‹time›: ‹reason›", plus "last successful upload" shown per rule
A gap of missed photos Picked up by the next periodic rescan, since they aren't in the log. "Scan now" makes it immediate
One file keeps failing Marked failed with a reason, doesn't block the queue, Retry / Ignore

Describe the solution you'd like

Scope: upload only. No two-way sync and no syncing of deletions (that's #17570). Local files are only changed by the existing "original file" action.

Principles:

  1. A file is uploaded once per account, not once per folder rule. Re-adding or editing a rule never re-uploads a file that was already uploaded.
  2. Files that are already on the server are always skipped. Skip / Rename / Overwrite / Ask only apply when the existing file is different (same name, different content).
  3. Every outcome is visible. Each file ends in a named state. Stalls and permanent failures produce a notification, never silence.
  4. Recovery is a scan, not a reset. Users never need to delete a rule or clear app data to get uploads going again.

Per-file decision:

candidate (in rule folder, captured inside the "upload from" window - see below)
 ├─ in this account's upload log? ──────────── yes → skip
 └─ target path free on server? ────────────── yes → upload, then log it
     └─ exists: identical? ─────────────────── yes → skip, log it
                 different → collision policy (default: Ask)

I suggest changing a few settings slightly, not adding a lot of new ones:

  • "Also upload existing files" becomes "Upload files from: All / ‹date›", compared against the capture date. It's never reset implicitly.
  • The collision setting is labelled "If a different file with the same name exists".

Recovery: one new menu entry, "Scan now…", with a From date and a mode:

  • This idea is based on #17462 but instead of a simple button, I think we should add a date-picker and mode-selector (implemented in 2 separate increments below to not block the current work):
    • Re-scan (default): respects the upload log. Safe to run any time.
    • Missing: ignores the log and skips files identical on the server. Warns that moved or deleted files will come back.
    • All: ignores the log and uploads everything within the window (conflicts resolved according to settings).

The time window:

Today, the only options for selecting files to consider for upload is "All" ("Upload existing files" enabled) or "From the config was saved" (The timestamp the app uses when "Upload existing files" are disabled.
My idea is to introduce a simple way to set a time-window to select which files to consider for uploads.

If uploads have failed for a period of time, one might want to resume uploads from a known date when the you know the upload last worked without worrying that hundreds of old files are re-uploaded. Or you can manually stop uploads for a period, and then re-enable them later, and decide if you want to upload all or only parts of the backlog.

The improved upload log:

My idea is an upload log that is computed on the device and keyed by content hash, so it works with any server. Server-side checksums (nextcloud/server#11138, nextcloud/server#56057) would improve the identity check but are not a prerequisite. The server already accepts OC-Checksum (nextcloud/server#51729).

Increments

I have split this up in 6 increments. Increments 1, 2 and 3 are independent of each other and work with what the app has today. Increments 4–6 build on each other.

  • 1. Visibility: "last successful upload" per rule, a stall notification, and failed files that don't block the queue. No server changes needed. Relates to #8154 ("Last auto upload ran at xx:yy"), #14233, #16639, #16207, #260. The battery-saver banner (#15717) is the pattern to reuse.
    Depends on: nothing.
  • 2. Always skip identical files, under every collision policy, and send OC-Checksum on upload. Relates to #8948, #4255, #10194, #12861, and the earlier attempt #16231.
    Depends on: nothing. Uses checksums where the server has them, and today's metadata comparison otherwise.
  • 3. "Scan now…" with a From date, Re-scan mode only. Relates to #15782, #15051, #15887 and draft #17462.
    Depends on: nothing. It respects the app's existing record of uploaded files, and the date limits any re-upload to the chosen window if that record has been lost.
  • 4. A per-file upload log keyed by content hash, kept across rule changes, and covering files that were skipped. Relates to #12967 (and #17522).
    Depends on: nothing, but it's the foundation for 5 and 6.
  • 5. "Scan now…" modes Missing and All. They're added to the dialog from increment 3.
    Depends on: 3 and 4. Both modes are defined as "ignore the log", so they need a reliable log to ignore.
  • 6. A server copy of the log, restored on a new device or reinstall. Android backup is disabled on purpose (see the maintainer comments in #11090), so a per-device log file in the user's own Nextcloud is used instead. Relates to #11090 and #6911.
    Depends on: 4.

Describe alternatives you've considered

  • A plain "force upload" button (#15887, #17462): without a date window and a reliable log, it either re-uploads everything or misses the gap.
  • Relying on server presence only: it can't know about files that were moved or deleted after upload.
  • Android backup or migration of the app database: it's disabled, many users avoid Google backup, and migration tools change DATE_ADDED, mtimes and paths.

Additional context

I'm a self-hosting user. Our family's phones auto-upload to private folders, and we then move the keepers into a shared folder. That's how we ran into these problems. I'm not an Android developer, so I can't vouch for code changes myself. I can help with testing builds, collecting logs, and refining the expected behaviour above. The code reading behind this was done with AI assistance (Claude); I checked the conclusions against the tracker.

主要言語
Kotlin
スター
5.7k
フォーク
2.1k
平均マージ
1日 22時間
マージ済み PR(30日)
135

環境構築

Codespaces で開く

このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

nextcloud/android のほかの issue

nextcloud/android の issue をすべて見る

似ている issue

Kotlin の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。