Unicode patch email rejected

まず parsemail 管理コマンドと patchwork/parser.py:1191 の parse_mail 呼び出しから始め、次に models.py の Patch.save を追跡します。報告された Unicode メールの失敗を再現し、パッチの内容が MySQL ベースのデータベースにどのように保存されるかを調べます。報告された OperationalError を発生させずにパッチが解析・保存されれば完了です。

This patch on libc-alpha did not appear in patchwork as it crashed during parsing:

https://sourceware.org/pipermail/libc-alpha/2022-October/142506.html

Here's the backtrace at the point of the crash:

Error when parsing incoming email: OperationalError(1366, "Incorrect string value: '\\xF0\\x91\\x82\\x80 0...' for column `patchwork`.`patchwork_patch`.`content` at row 1")
Traceback (most recent call last):
  File "/path/to/patchwork/lib/python3/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/path/to/patchwork/lib/python3/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/path/to/patchwork/lib/python3/lib64/python3.6/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/path/to/patchwork/lib/python3/lib64/python3.6/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/path/to/patchwork/lib/python3/lib64/python3.6/site-packages/MySQLdb/connections.py", line 254, in query
    _mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (1366, "Incorrect string value: '\\xF0\\x91\\x82\\x80 0...' for column `patchwork`.`patchwork_patch`.`content` at row 1")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/path/to/patchwork/patchwork.sourceware.org/patchwork/management/commands/parsemail.py", line 60, in handle
    result = parse_mail(mail, options['list_id'])
  File "/path/to/patchwork/patchwork.sourceware.org/patchwork/parser.py", line 1191, in parse_mail
    state=find_state(mail),
  File "/path/to/patchwork/lib/python3/lib/python3.6/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/path/to/patchwork/lib/python3/lib/python3.6/site-packages/django/db/models/query.py", line 447, in create
    obj.save(force_insert=True, using=self.db)
  File "/path/to/patchwork/patchwork.sourceware.org/patchwork/models.py", line 554, in save
    super(Patch, self).save(**kwargs)
  File "/path/to/patchwork/patchwork.sourceware.org/patchwork/models.py", line 385, in save
    super(EmailMixin, self).save(*args, **kwargs)
  File "/path/to/patchwork/lib/python3/lib/python3.6/site-packages/django/db/models/base.py", line 754, in save
    force_update=force_update, update_fields=update_fields)
  File "/path/to/patchwork/lib/python3/lib/python3.6/site-packages/django/db/models/base.py", line 792, in save_base
    force_update, using, update_fields,
  File "/path/to/patchwork/lib/python3/lib/python3.6/site-packages/django/db/models/base.py", line 895, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/path/to/patchwork/lib/python3/lib/python3.6/site-packages/django/db/models/base.py", line 935, in _do_insert
    using=using, raw=raw,
  File "/path/to/patchwork/lib/python3/lib/python3.6/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/path/to/patchwork/lib/python3/lib/python3.6/site-packages/django/db/models/query.py", line 1254, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/path/to/patchwork/lib/python3/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1397, in execute_sql
    cursor.execute(sql, params)
  File "/path/to/patchwork/lib/python3/lib/python3.6/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/path/to/patchwork/lib/python3/lib/python3.6/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/path/to/patchwork/lib/python3/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/path/to/patchwork/lib/python3/lib/python3.6/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/path/to/patchwork/lib/python3/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/path/to/patchwork/lib/python3/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/path/to/patchwork/lib/python3/lib64/python3.6/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/path/to/patchwork/lib/python3/lib64/python3.6/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/path/to/patchwork/lib/python3/lib64/python3.6/site-packages/MySQLdb/connections.py", line 254, in query
    _mysql.connection.query(self, query)
django.db.utils.OperationalError: (1366, "Incorrect string value: '\\xF0\\x91\\x82\\x80 0...' for column `patchwork`.`patchwork_patch`.`content` at row 1")

The backtrace was attained using the following change to the sourceware instance, maybe something like this could be useful for general purpose logging too:

 
 PYTHONPATH="${PATCHWORK_BASE}:${PATCHWORK_BASE}/lib/python:$PYTHONPATH" \
     DJANGO_SETTINGS_MODULE="$DJANGO_SETTINGS_MODULE" \
-    "$PW_PYTHON" "$PATCHWORK_BASE/manage.py" parsemail "$@"
+    "$PW_PYTHON" "$PATCHWORK_BASE/manage.py" parsemail \
+    "$@" >> $PATCHWORK_BASE/patchwork-mailstat.log 2>&1
 
 # NOTE(stephenfin): We must return 0 here. When parsemail is used as a
 # delivery command from a mail server like postfix (as it is intended