S3: Updating mod-time of file wipes other headers on original file · rclone/rclone#5243

(4 comments) (1 reaction) (0 assignees)Go (5,090 forks)batch import

Remote: S3bughelp wanted

Repository metrics

Stars: (57,185 stars)
PR merge metrics: (Avg merge 32d 20h) (23 merged PRs in 30d)

Description

What is the problem you are having with rclone?

I'm using the S3 backend, and if rclone encounters an identical file in the destination, but the source's timestamp is newer, by default it tries to update the modification time of the destination file. However, this process seems to wipe other headers that might be stored on the file, which doesn't seem like it's expected (or at least I can't find any documentation on). For example, Cache-Control or Content-Encoding headers can get wiped.

It looks like you can workaround this issue by disabling the modification update process (--no-update-modtime), or by using the checksum syncing mechanism (--checksum). But this default behavior still seems like it's perhaps unexpected, since it can lead to things ending up in a funky state (for example, if the Content-Encoding header gets wiped, then the data being served up by the bucket may no longer match what the actual content encoding of the file is).

What is your rclone version (output from `rclone version`)

rclone v1.55.0
- os/type: darwin
- os/arch: amd64
- go/version: go1.16.2
- go/linking: dynamic
- go/tags: none

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

macOS Catalina, 64 bit

Which cloud storage system are you using? (e.g. Google Drive)

AWS S3

The command you were trying to run (e.g. `rclone copy /tmp remote:tmp`)

rclone copy -vv --dump requests --s3-no-check-bucket --header-upload "Content-Encoding: gzip" --header-upload "Cache-Control: max-age=60" ./test.txt :s3:my-test-bucket/

A log from the command with the `-vv` flag (e.g. output from `rclone -vv copy /tmp remote:tmp`)

Create a local file:
```
touch test.txt`
```

The initial copy of the new file works as expected:

rclone copy -vv --dump requests --s3-no-check-bucket --header-upload "Content-Encoding: gzip" --header-upload "Cache-Control: max-age=60" ./test.txt :s3:my-test-bucket/

2021/04/13 10:06:08 NOTICE: Config file "/Users/nmuerdte/.config/rclone/rclone.conf" not found - using defaults
2021/04/13 10:06:08 DEBUG : rclone: Version "v1.55.0" starting with parameters ["rclone" "copy" "-vv" "--dump" "requests" "--s3-no-check-bucket" "--header-upload" "Content-Encoding: gzip" "--header-upload" "Cache-Control: max-age=60" "./test.txt" ":s3:my-test-bucket/"]
2021/04/13 10:06:08 DEBUG : Creating backend with remote "./test.txt"
2021/04/13 10:06:08 DEBUG : fs cache: adding new entry for parent of "./test.txt", "/tmp"
2021/04/13 10:06:08 DEBUG : Creating backend with remote ":s3:my-test-bucket/"
2021/04/13 10:06:08 DEBUG : :s3: detected overridden config - adding "{p8NAi}" suffix to name
2021/04/13 10:06:08 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/04/13 10:06:08 DEBUG : fs cache: renaming cache item ":s3:my-test-bucket/" to be canonical ":s3{p8NAi}:my-test-bucket"
2021/04/13 10:06:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/13 10:06:08 DEBUG : HTTP REQUEST (req 0xc000170200)
2021/04/13 10:06:08 DEBUG : HEAD /my-test-bucket/test.txt HTTP/1.1
Host: s3.us-east-2.amazonaws.com
User-Agent: rclone/v1.55.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210413T160608Z

2021/04/13 10:06:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/13 10:06:08 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/13 10:06:08 DEBUG : HTTP RESPONSE (req 0xc000170200)
2021/04/13 10:06:08 DEBUG : HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Tue, 13 Apr 2021 16:06:08 GMT
Server: AmazonS3
X-Amz-Delete-Marker: true
X-Amz-Id-2: hRtSP+MCvNjGzlMdERQZblLbjwMD2K9FpTpawySGZdmf9a8gkQmcjsEO9Oj40ty6uqjdlgtA+B0=
X-Amz-Request-Id: TMKQR9ZX61TSH4P7
X-Amz-Version-Id: f8og4Pmgizd8x7QXRrcCQtHuTYvIW6l.

2021/04/13 10:06:08 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/13 10:06:08 DEBUG : test.txt: Need to transfer - File not found at Destination
2021/04/13 10:06:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/13 10:06:08 DEBUG : HTTP REQUEST (req 0xc00091c300)
2021/04/13 10:06:08 DEBUG : PUT /my-test-bucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=XXXX%2F20210413%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20210413T160608Z&X-Amz-Expires=900&X-Amz-SignedHeaders=cache-control%3Bcontent-encoding%3Bcontent-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime&X-Amz-Signature=XXXX HTTP/1.1
Host: s3.us-east-2.amazonaws.com
User-Agent: rclone/v1.55.0
Content-Length: 0
cache-control: max-age=60
content-encoding: gzip
content-md5: 1B2M2Y8AsgTpgAmY7PhCfg==
content-type: text/plain; charset=utf-8
x-amz-acl: private
x-amz-meta-mtime: 1618329958.152330531
Accept-Encoding: gzip

2021/04/13 10:06:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/13 10:06:08 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/13 10:06:08 DEBUG : HTTP RESPONSE (req 0xc00091c300)
2021/04/13 10:06:08 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Tue, 13 Apr 2021 16:06:09 GMT
Etag: "d41d8cd98f00b204e9800998ecf8427e"
Server: AmazonS3
X-Amz-Id-2: UroKjvjc8QMqtL4xiKF91Iwiou3r0pXHpX1x99BqfFUszYZ2VdhwdRO65GolqSb2yGD0ml+Vv58=
X-Amz-Request-Id: TMKM2FZ3G3Y3JA2N
X-Amz-Version-Id: CVlOvUYEQfcwj65MZz_hRZhKoCSqOcCH

2021/04/13 10:06:08 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/13 10:06:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/13 10:06:08 DEBUG : HTTP REQUEST (req 0xc000170800)
2021/04/13 10:06:08 DEBUG : HEAD /my-test-bucket/test.txt HTTP/1.1
Host: s3.us-east-2.amazonaws.com
User-Agent: rclone/v1.55.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210413T160608Z

2021/04/13 10:06:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/13 10:06:08 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/13 10:06:08 DEBUG : HTTP RESPONSE (req 0xc000170800)
2021/04/13 10:06:08 DEBUG : HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: max-age=60
Content-Encoding: gzip
Content-Type: text/plain; charset=utf-8
Date: Tue, 13 Apr 2021 16:06:09 GMT
Etag: "d41d8cd98f00b204e9800998ecf8427e"
Last-Modified: Tue, 13 Apr 2021 16:06:09 GMT
Server: AmazonS3
X-Amz-Id-2: E0a1x6Kwr3D9Jr3dN3Uk552UkR8Q4rgGpK8rK10Zfo2J5kFFH3OynYjzsf3To3iTPpaYxNAS8Og=
X-Amz-Meta-Mtime: 1618329958.152330531
X-Amz-Request-Id: TMKJP629X0QDD2BA
X-Amz-Version-Id: CVlOvUYEQfcwj65MZz_hRZhKoCSqOcCH
Content-Length: 0

2021/04/13 10:06:08 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/13 10:06:08 DEBUG : test.txt: MD5 = d41d8cd98f00b204e9800998ecf8427e OK
2021/04/13 10:06:08 INFO  : test.txt: Copied (new)
2021/04/13 10:06:08 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.6s

2021/04/13 10:06:08 DEBUG : 6 go routines active

Verifying that the Cache-Control and Content-Encoding headers were stored as expected:

aws s3api head-object --bucket my-test-bucket --key test.txt

{
    "AcceptRanges": "bytes",
    "LastModified": "2021-04-13T16:06:09+00:00",
    "ContentLength": 0,
    "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
    "VersionId": "CVlOvUYEQfcwj65MZz_hRZhKoCSqOcCH",
    "CacheControl": "max-age=60",
    "ContentEncoding": "gzip",
    "ContentType": "text/plain; charset=utf-8",
    "Metadata": {
        "mtime": "1618329958.152330531"
    }
}

Update the timestamp of the local file:
```
touch test.txt`
```

Trigger another copy, which this time will just update the modified time on the file:

rclone copy -vv --dump requests --s3-no-check-bucket --header-upload "Content-Encoding: gzip" --header-upload "Cache-Control: max-age=60" ./test.txt :s3:my-test-bucket/

2021/04/13 10:06:59 NOTICE: Config file "/Users/nmuerdte/.config/rclone/rclone.conf" not found - using defaults
2021/04/13 10:06:59 DEBUG : rclone: Version "v1.55.0" starting with parameters ["rclone" "copy" "-vv" "--dump" "requests" "--s3-no-check-bucket" "--header-upload" "Content-Encoding: gzip" "--header-upload" "Cache-Control: max-age=60" "./test.txt" ":s3:my-test-bucket/"]
2021/04/13 10:06:59 DEBUG : Creating backend with remote "./test.txt"
2021/04/13 10:06:59 DEBUG : fs cache: adding new entry for parent of "./test.txt", "/tmp"
2021/04/13 10:06:59 DEBUG : Creating backend with remote ":s3:my-test-bucket/"
2021/04/13 10:06:59 DEBUG : :s3: detected overridden config - adding "{p8NAi}" suffix to name
2021/04/13 10:06:59 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/04/13 10:06:59 DEBUG : fs cache: renaming cache item ":s3:my-test-bucket/" to be canonical ":s3{p8NAi}:my-test-bucket"
2021/04/13 10:06:59 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/13 10:06:59 DEBUG : HTTP REQUEST (req 0xc00071a200)
2021/04/13 10:06:59 DEBUG : HEAD /my-test-bucket/test.txt HTTP/1.1
Host: s3.us-east-2.amazonaws.com
User-Agent: rclone/v1.55.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210413T160659Z

2021/04/13 10:06:59 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/13 10:07:00 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/13 10:07:00 DEBUG : HTTP RESPONSE (req 0xc00071a200)
2021/04/13 10:07:00 DEBUG : HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: max-age=60
Content-Encoding: gzip
Content-Type: text/plain; charset=utf-8
Date: Tue, 13 Apr 2021 16:07:01 GMT
Etag: "d41d8cd98f00b204e9800998ecf8427e"
Last-Modified: Tue, 13 Apr 2021 16:06:09 GMT
Server: AmazonS3
X-Amz-Id-2: n+EwHbc+4hmcIzQUqKcifIhkGMg7qoNju/Yp0dg7YXcw/xr27QdehjKQcmeCTNFGCGYWHC1XlAU=
X-Amz-Meta-Mtime: 1618329958.152330531
X-Amz-Request-Id: A118XNQ9WVN58ND3
X-Amz-Version-Id: CVlOvUYEQfcwj65MZz_hRZhKoCSqOcCH
Content-Length: 0

2021/04/13 10:07:00 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/13 10:07:00 DEBUG : test.txt: Modification times differ by -57.150712469s: 2021-04-13 10:06:55.303043 -0600 MDT, 2021-04-13 10:05:58.152330531 -0600 MDT
2021/04/13 10:07:00 DEBUG : test.txt: MD5 = d41d8cd98f00b204e9800998ecf8427e OK
2021/04/13 10:07:00 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/13 10:07:00 DEBUG : HTTP REQUEST (req 0xc0006dd400)
2021/04/13 10:07:00 DEBUG : PUT /my-test-bucket/test.txt HTTP/1.1
Host: s3.us-east-2.amazonaws.com
User-Agent: rclone/v1.55.0
Content-Length: 0
Authorization: XXXX
Content-Type: text/plain; charset=utf-8
X-Amz-Acl: private
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Copy-Source: my-test-bucket/test.txt
X-Amz-Date: 20210413T160700Z
X-Amz-Meta-Mtime: 1618330015.303043
X-Amz-Metadata-Directive: REPLACE
Accept-Encoding: gzip

2021/04/13 10:07:00 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/13 10:07:00 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/13 10:07:00 DEBUG : HTTP RESPONSE (req 0xc0006dd400)
2021/04/13 10:07:00 DEBUG : HTTP/1.1 200 OK
Content-Length: 234
Content-Type: application/xml
Date: Tue, 13 Apr 2021 16:07:01 GMT
Server: AmazonS3
X-Amz-Copy-Source-Version-Id: CVlOvUYEQfcwj65MZz_hRZhKoCSqOcCH
X-Amz-Id-2: bAS+UL9u517emrdA/zXgrlgPg88vUWO9ZDDOMAFai6guSwzWkSQIO+L91l01+WCaXxj8JWAw3GU=
X-Amz-Request-Id: A1118PR5M9E11ARA
X-Amz-Version-Id: zUkZ49sdQL1XcnYsnTpLshCwaoGMb1yZ

2021/04/13 10:07:00 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/13 10:07:00 INFO  : test.txt: Updated modification time in destination
2021/04/13 10:07:00 DEBUG : test.txt: Unchanged skipping
2021/04/13 10:07:00 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.5s

2021/04/13 10:07:00 DEBUG : 6 go routines active

This results in the Cache-Control and Content-Encoding headers being removed from the stored file:

aws s3api head-object --bucket my-test-bucket --key test.txt

{
    "AcceptRanges": "bytes",
    "LastModified": "2021-04-13T16:07:01+00:00",
    "ContentLength": 0,
    "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
    "VersionId": "zUkZ49sdQL1XcnYsnTpLshCwaoGMb1yZ",
    "ContentType": "text/plain; charset=utf-8",
    "Metadata": {
        "mtime": "1618330015.303043"
    }
}

Contributor guide

Research direction: Inspect the S3 backend's updateModTime function in rclone. Trace how it copies metadata when updating modification time. Identify where headers like Cache Control and Content Encoding are lost.
Tech stack: go
Domain: backendapicloud
Issue type: Bug
Difficulty: 3
Estimated time: 1-3 hours
Activity status: Active
Clarity: Clear
Prerequisites: GitGo
Newbie friendliness: 70

Repository metrics

Description

What is the problem you are having with rclone?

What is your rclone version (output from rclone version)

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

Which cloud storage system are you using? (e.g. Google Drive)

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)

A log from the command with the -vv flag (e.g. output from rclone -vv copy /tmp remote:tmp)

Contributor guide

Get fresh easy issues in your inbox.

What is your rclone version (output from `rclone version`)

The command you were trying to run (e.g. `rclone copy /tmp remote:tmp`)

A log from the command with the `-vv` flag (e.g. output from `rclone -vv copy /tmp remote:tmp`)