One rejected last_error can still roll back a whole batch (MySQL non-utf8mb4 charset; per-row update fallback)

オープン
  • 何か問題がありますか？
#141 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
35/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
kotlin, mysql, postgresql
領域
backend, database

調査の方向性

Start with OutboxProcessor.processNext, both store implementations, and 001__create_okapi_outbox_table.sql; compare the batch update transaction behavior with the two proposed options. Decide how row-level failures should preserve the other results, then add the latin1 MySQL test with one healthy entry and one emoji error. Done when the healthy entry remains DELIVERED and the rejected row follows the agreed retry behavior.

索引モデルが issue の本文から書いたものです。

説明

bug

Area: okapi-core OutboxProcessor, both stores, MySQL schema · Follow-up to: #130 / #140

Problem

OutboxProcessor.processNext still writes every result of a batch with one updateAfterProcessingBatch call inside the claim transaction. #140 removes the two known values the database rejected in last_error: values over 64 KB on MySQL, and NUL characters on Postgres. But any other rejected value in one row still rolls back the whole batch. Entries that were already delivered go back to PENDING with retries unchanged and are redelivered on every tick.

One more cheap trigger remains on MySQL. 001__create_okapi_outbox_table.sql creates okapi_outbox without an explicit CHARACTER SET, so the table inherits the schema or server default. On a schema that is still latin1 or utf8mb3, which is common on databases from the 5.7 era or migrated ones, a last_error with an emoji or any other character outside that charset fails in strict mode with ERROR 1366 Incorrect string value. The same applies to an HTTP error body or an exception message. The whole batch then rolls back again, delivered entries included. (I haven't reproduced this yet; MySQL 8 defaults to utf8mb4, which is why the tests don't hit it.)

Options

  1. Isolate per-row update failures (covers every cause). If the batch update throws, fall back to updating row by row. A row the database still rejects is written with a sanitized placeholder error, or skipped and left for the next tick with retries advanced, so the other rows' results survive. On Postgres this needs a savepoint per row, because an error aborts the transaction. This changes the store/processor contract, so it needs a design decision.
  2. MySQL charset. Add DEFAULT CHARSET=utf8mb4 for new installs, plus a changeset that converts existing tables (ALTER TABLE okapi_outbox CONVERT TO CHARACTER SET utf8mb4). The conversion rebuilds the table, so the rollout notes need to cover it. A cheaper, narrower option is to convert only last_error.
  3. Sanitize to the column charset. Replace characters the column can't store before writing. This is fragile, because the store would need to know the column charset.

Option 1 fixes the class of problem; option 2 removes the known trigger. I'd do both, starting with option 2 because it is smaller.

Tests

  • MySQL with a latin1 schema: a batch with one healthy entry and one whose error contains an emoji. The healthy entry stays DELIVERED.
  • For option 1: a store that rejects one row of a batch. The rest of the batch keeps its results, and the rejected row's retries advances.
主要言語
Kotlin
スター
14
フォーク
0
平均マージ
9時間 54分
マージ済み PR（30日）
11

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

softwaremill/okapi のほかの issue

softwaremill/okapi の issue をすべて見る

似ている issue

Kotlin の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。