Too many duplicate key value violates unique constraint errors.

Open
#173 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
30/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
postgresql, python
Domain
databases

Research direction

Start with crypto/store/asyncpg/store.py at put_devices and state_store/asyncpg/store.py at set_members, then trace their copy_records_to_table calls in the duplicate-key traces. Reproduce the crypto_device and mx_user_profile violations with multiple bot instances and determine the expected handling for existing rows. Done means these recurring errors are handled without losing the stored device or membership data.

Written by the indexing model from the issue text.

Description

I have logged into multiple bot accounts clients in maubot and created multiple instances for each account, relying on different plugins,
there are a lot of errors in the logs.
Although it seems that the bot is running normally, there are a lot of errors in the logs.
The logs look like this:


Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/mautrix/client/syncer.py", line 236, in _catch_errors
    await handler(data)
  File "/usr/lib/python3.11/site-packages/mautrix/crypto/machine.py", line 171, in handle_device_lists
    await self._fetch_keys(device_lists.changed, include_untracked=False)
  File "/usr/lib/python3.11/site-packages/mautrix/crypto/device_lists.py", line 80, in _fetch_keys
    await self.crypto_store.put_devices(user_id, new_devices)
  File "/usr/lib/python3.11/site-packages/mautrix/crypto/store/asyncpg/store.py", line 615, in put_devices
    await conn.copy_records_to_table("crypto_device", records=data, columns=columns)
  File "/usr/lib/python3.11/site-packages/mautrix/util/async_db/connection.py", line 38, in wrapper
    ret = await func(self, arg, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/mautrix/util/async_db/connection.py", line 151, in copy_records_to_table
    return await self.wrapped.copy_records_to_table(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/asyncpg/connection.py", line 983, in copy_records_to_table
    return await self._protocol.copy_in(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "asyncpg/protocol/protocol.pyx", line 529, in copy_in
asyncpg.exceptions.UniqueViolationError: duplicate key value violates unique constraint "crypto_device_pkey"
DETAIL:  Key (user_id, device_id)=(@name:domain.com, cliendid) 
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/mautrix/client/syncer.py", line 236, in _catch_errors
    await handler(data)
  File "/usr/lib/python3.11/site-packages/mautrix/crypto/machine.py", line 171, in handle_device_lists
    await self._fetch_keys(device_lists.changed, include_untracked=False)
  File "/usr/lib/python3.11/site-packages/mautrix/crypto/device_lists.py", line 80, in _fetch_keys
    await self.crypto_store.put_devices(user_id, new_devices)
  File "/usr/lib/python3.11/site-packages/mautrix/crypto/store/asyncpg/store.py", line 615, in put_devices
    await conn.copy_records_to_table("crypto_device", records=data, columns=columns)
  File "/usr/lib/python3.11/site-packages/mautrix/util/async_db/connection.py", line 38, in wrapper
    ret = await func(self, arg, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/mautrix/util/async_db/connection.py", line 151, in copy_records_to_table
    return await self.wrapped.copy_records_to_table(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/asyncpg/connection.py", line 983, in copy_records_to_table
    return await self._protocol.copy_in(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "asyncpg/protocol/protocol.pyx", line 529, in copy_in
asyncpg.exceptions.UniqueViolationError: duplicate key value violates unique constraint "crypto_device_pkey"
DETAIL:  Key (user_id, device_id)=(@name:domain.com, cliendid) already exists.

Sometimes very similar violates unique constraint errors appear but with different duplicate keys, such as mx_user_profile_pkey.

Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/mautrix/client/syncer.py", line 236, in _catch_errors
    await handler(data)
  File "/data/plugins/org.jobmachine.chatgpt-v0.0.10.mbp/gpt.py", line 87, in on_message
    if not await self.should_respond(event):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/plugins/org.jobmachine.chatgpt-v0.0.10.mbp/gpt.py", line 67, in should_respond
    if len(await self.client.get_joined_members(event.room_id)) == 2:
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/mautrix/client/store_updater.py", line 218, in get_joined_members
    await self.state_store.set_members(room_id, members, only_membership=Membership.JOIN)
  File "/usr/lib/python3.11/site-packages/mautrix/client/state_store/asyncpg/store.py", line 169, in set_members
    await conn.copy_records_to_table(
  File "/usr/lib/python3.11/site-packages/mautrix/util/async_db/connection.py", line 38, in wrapper
    ret = await func(self, arg, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/mautrix/util/async_db/connection.py", line 151, in copy_records_to_table
    return await self.wrapped.copy_records_to_table(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/asyncpg/connection.py", line 983, in copy_records_to_table
    return await self._protocol.copy_in(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "asyncpg/protocol/protocol.pyx", line 529, in copy_in
asyncpg.exceptions.UniqueViolationError: duplicate key value violates unique constraint "mx_user_profile_pkey"
DETAIL:  Key (room_id, user_id)=(!xxxxxx:domain.com, @name:domain.com) already exists.

Maybe it's related to: https://github.com/mautrix/python/issues/160

Dominant language
Python
Stars
249
Forks
84
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from mautrix/python

All issues in mautrix/python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.