Fix deadlocks caused by EctoOrdered

Bắt đầu với các stack frame của EctoOrdered, đặc biệt là ecto_ordered.ex:242 và :172, sau đó lần theo luồng sync gọi qua code_corps/github/sync/issue/task/task.ex và sync.ex. Tái hiện kịch bản sync GitHub đồng thời và xác định cách cần xác minh việc hoàn tất bằng cách loại bỏ các deadlock PostgreSQL đã được báo cáo.

Problem

Our new sync process revealed some issues caused in our use of EctoOrdered when there are a large number of writes. At the kind of scale we need for highly concurrent GitHub syncing, we are hitting deadlocks in the database.

Here's an example of the errors:

2017-11-02T23:39:42.632335Z [error] Task #PID<0.22638.6> started from #PID<0.3430.0> terminating
** (Postgrex.Error) ERROR 40P01 (deadlock_detected): deadlock detected

Process 14260 waits for ShareLock on transaction 1981764; blocked by process 14258.
Process 14258 waits for ShareLock on transaction 1981765; blocked by process 14260.
    (ecto) lib/ecto/adapters/sql.ex:440: Ecto.Adapters.SQL.execute_or_reset/7
    (ecto_ordered) lib/ecto_ordered.ex:242: EctoOrdered.increment_other_ranks/2
    (ecto_ordered) lib/ecto_ordered.ex:172: EctoOrdered.ensure_unique_position/2
    (ecto) lib/ecto/repo/schema.ex:456: anonymous fn/2 in Ecto.Repo.Schema.run_prepare/2
    (elixir) lib/enum.ex:1811: Enum."-reduce/3-lists^foldl/2-0-"/3
    (ecto) lib/ecto/repo/schema.ex:268: anonymous fn/14 in Ecto.Repo.Schema.do_update/4
    (ecto) lib/ecto/repo/schema.ex:768: anonymous fn/3 in Ecto.Repo.Schema.wrap_in_transaction/6
    (ecto) lib/ecto/adapters/sql.ex:576: anonymous fn/3 in Ecto.Adapters.SQL.do_transaction/3
    (db_connection) lib/db_connection.ex:1275: DBConnection.transaction_run/4
    (db_connection) lib/db_connection.ex:1199: DBConnection.run_begin/3
    (db_connection) lib/db_connection.ex:790: DBConnection.transaction/3
    (code_corps) lib/code_corps/github/sync/issue/task/task.ex:51: CodeCorps.GitHub.Sync.Issue.Task.find_or_create_task/2
    (elixir) lib/enum.ex:1255: Enum."-map/2-lists^map/1-0-"/2
    (elixir) lib/enum.ex:1255: Enum."-map/2-lists^map/1-0-"/2
    (code_corps) lib/code_corps/github/sync/issue/task/task.ex:45: CodeCorps.GitHub.Sync.Issue.Task.sync_project_github_repo/1
    (code_corps) lib/code_corps/github/sync/sync.ex:178: CodeCorps.GitHub.Sync.sync_project_github_repo/1
    (elixir) lib/task/supervised.ex:85: Task.Supervised.do_apply/2
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Function: #Function<3.4867021/0 in CodeCorpsWeb.ProjectGithubRepoController.create/2>
    Args: []

One approach to this is to rewrite our card syncing to use a single position column with floats and some ordering scheme like this one, with most of the operations happening on the server itself.