Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

dyno failing to restore incremental backup?

Open
  • Is something wrong?
#85 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
aws, javascript, nodejs
Domain
databases

Research direction

Reproduce the pipeline in the issue with bin/incremental-backfill.js, bin/incremental-snapshot.js, and s3print. Read node_modules/dyno/lib/serialization.js and node_modules/dyno/bin/cli.js where the JSON parse failure occurs, then compare the snapshot output with dyno's expected input format. Done means the generated snapshot can be piped through dyno put successfully and the restored table can be compared with the source.

Written by the indexing model from the issue text.

Description

simple test case with a small table:

ENV=dev
TABLE=dsrtest2

. config.env.$ENV

bin/incremental-backfill.js $AWS_REGION/$TABLE s3://$BackupBucket/$BackupPrefix

bin/incremental-snapshot.js s3://$BackupBucket/$BackupPrefix/$TABLE  s3://$BackupBucket/${TABLE}-snapshot

s3print s3://$BackupBucket/${TABLE}-snapshot | dyno put $AWS_REGION/dsr-test-restore-$TABLE


%% sh test-backup.sh
12 - 11.89/s[Fri, 09 Dec 2016 23:54:59 GMT] [info] [incremental-snapshot] Starting snapshot from s3://dsr-ddb-rep-testing/testprefix/dsrtest2 to s3://dsr-ddb-rep-testing/dsrtest2-snapshot
[Fri, 09 Dec 2016 23:55:01 GMT] [info] [incremental-snapshot] Starting upload of part #0, 0 bytes uploaded, 12 items uploaded @ 6.26 items/s
[Fri, 09 Dec 2016 23:55:01 GMT] [info] [incremental-snapshot] Uploaded snapshot to s3://dsr-ddb-rep-testing/dsrtest2-snapshot
[Fri, 09 Dec 2016 23:55:01 GMT] [info] [incremental-snapshot] Wrote 12 items and 148 bytes to snapshot
undefined:1
�
^

SyntaxError: Unexpected token  in JSON at position 0
    at Object.parse (native)
    at Function.module.exports.deserialize (/Users/draistrick/git/github/dynamodb-replicator/node_modules/dyno/lib/serialization.js:49:18)
    at Transform.Parser.parser._transform (/Users/draistrick/git/github/dynamodb-replicator/node_modules/dyno/bin/cli.js:94:25)
    at Transform._read (_stream_transform.js:167:10)
    at Transform._write (_stream_transform.js:155:12)
    at doWrite (_stream_writable.js:307:12)
    at writeOrBuffer (_stream_writable.js:293:5)
    at Transform.Writable.write (_stream_writable.js:220:11)
    at Stream.ondata (stream.js:31:26)
    at emitOne (events.js:96:13)

Next step would be to diff the two tables - but the pipe to dyno fails. I've tried 1.0.0 and 1.3.0 with the same result.

What data format is dyno expecting? The file on s3 (tried multiple tables including real data tables) is a binary blob?

cheese:~%% aws --region=us-west-2 s3 cp s3://dsr-ddb-rep-testing/dsrtest-snapshot -
m�1�
��ߠl�EG�EB�uL0\�Tuq�ݵ#������$L�6�/8�%Z�r�[d�p
���5h)��X�ֻ�j�ƪ�
 ۘ��&�WJ'❑��`�T�������􁒷
cheese:~%%

So maybe this is a problem with backfill? or I'm missing something? :)

2016-12-09 18:54:35 149 dsrtest-snapshot
2016-12-09 18:55:01 148 dsrtest2-snapshot
2016-12-09 18:37:20 1428 receipt_log_dev-01-snapshot
2016-12-09 18:53:15 13457328 showdownlive_dev-01-snapshot

Dominant language
JavaScript
Stars
131
Forks
46
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from mapbox/dynamodb-replicator

All issues in mapbox/dynamodb-replicator

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.