[Enhancement] Adjust VARCHAR byte length calculation to fully support UTF-8 (utf8mb4)
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 68/100
Research direction
Start in TypeConverter.java at the VARCHAR byte-length calculation and review how the current multiplier and 65533 limit are applied. Update the calculation to account for up to four UTF-8 bytes per character, then verify that long utf8mb4 values do not produce an undersized VARCHAR definition or exceed the Doris limit.
Written by the indexing model from the issue text.
Description
Search before asking
- I had searched in the issues and found no similar issues.
Description
The current logic in TypeConverter.java uses a multiplier of 3 to calculate the required byte length for the Doris VARCHAR type:
// Current implementation
return length * 3 > 65533
? DorisType.STRING
: String.format("%s(%s)", DorisType.VARCHAR, length * 3);
This assumes a maximum of 3 bytes per character, which is insufficient for the widely used utf8mb4 character set (common in MySQL/MariaDB and other sources). The utf8mb4 encoding supports the full range of Unicode characters (including emojis), requiring up to 4 bytes per character.
If a source column contains 4-byte characters, the calculated byte length may underestimate the required size, leading to:
Data truncation or corruption during the synchronization process.
Load failures with errors such as "data length exceeded" or "row size too large" when Doris enforces the byte limit.
Solution
This change updates the byte multiplier from 3 to 4 to safely accommodate the full utf8mb4 character set, ensuring the calculated byte length is always sufficient for the defined character length, thus guaranteeing data integrity and preventing sync failures.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
- Dominant language
- Java
- Stars
- 384
- Forks
- 279
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 8
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/doris-flink-connector
-
Release Note 26.3.0 Openrelease notes
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
apache/doris-flink-connector#699 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apache/doris-flink-connector#611 · 4 comments ·
-
Release Note 26.2.0 Openrelease notes
Difficulty 2/5 1-3 hours Newbie friendliness 45/100
apache/doris-flink-connector#691 ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
apache/doris-flink-connector#682 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 52/100
apache/doris-flink-connector#666 ·
All issues in apache/doris-flink-connector
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
infinispan/infinispan#18150 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
opensearch-project/k-NN#3597 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100