seconv: Shift-JIS and GBK files without a BOM are read as windows-1252

Open
  • Is something wrong?
#16,376 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
58/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
csharp
Domain
backend

Research direction

Start with src/libse/Common/LanguageAutoDetect.cs, especially DetectAnsiEncoding around line 2506 and the call from GetEncodingFromFile around line 2860. Compare how code page 949 is detected, then use the attached Shift-JIS, GBK, and cp949 samples to check encoding and language results with seconv info. Done means the Japanese and Chinese samples are identified correctly without changing the cp949 result.

Written by the indexing model from the issue text.

Description

Summary

Japanese and Chinese subtitle or lyric files in a legacy encoding (Shift-JIS, GBK) and without a BOM are decoded as windows-1252, so seconv translates and saves mojibake. libse's ANSI auto-detection tries Korean (cp949) but no Japanese or Chinese code page.

Seen with seconv 5.4.0-beta6 on Windows 11. The code is the same on main (cf1c3e3).

Steps to reproduce
  1. Unzip the attached seconv-encoding-samples.zip. shift-jis.lrc, gbk.lrc and cp949.lrc hold three lines of lyrics each, saved in that encoding without a BOM; utf-8.lrc is the Japanese file again in UTF-8, for comparison.
  2. seconv info shift-jis.lrc --json
  3. seconv shift-jis.lrc LrcLyrics --output-folder out
Expected and actual

seconv info on the attached files:

File Encoding encoding reported language reported Expected
shift-jis.lrc Shift-JIS (cp932) windows-1252 no shift_jis, ja
gbk.lrc GBK (cp936) windows-1252 sk gb2312, zh
cp949.lrc cp949 (Korean) euc-kr ko as reported

Step 3 writes the first line as [00:00.00]‚±‚ñ‚É‚¿‚ÍAŒ³‹C‚Å‚·‚© (UTF-8 with BOM). With --translate-to, that mojibake is what the translation engine receives, and the translated file replaces the original.

Side note: info prints zh without a script for UTF-8 Chinese too: SubtitleInfoGatherer calls AutoDetectGoogleLanguageOrNull without detectChineseScript, while the translation path passes true and gets zh-Hans / zh-Hant. I am not sure if this is intentional or a bug so figured I would call it out.

Cause

LanguageAutoDetect.GetEncodingFromFile finds no BOM, IsUtf8 fails, and with AutoGuessAnsiEncoding (default true) it calls DetectAnsiEncoding (LanguageAutoDetect.cs#L2860).

DetectAnsiEncoding tries 1251, 949, 1253, 1250, 1252, 1254, 28592, 28595, 874, 1255, 1256, 28591 and 28599 (LanguageAutoDetect.cs#L2506). There is no 932 (Shift-JIS), 936 (GBK) or 950 (Big5), so a Japanese or Chinese file can never be detected and falls through to the windows-1252 default.

Suggestion

Add 932, 936 and 950 as candidates in DetectAnsiEncoding, the way 949 is handled with AutoDetectWordsKorean: decode the buffer with each code page and count Hiragana/Katakana (U+3040–U+30FF, unambiguous for Japanese) or CJK ideographs (U+4E00–U+9FFF), and take the code page whose decode has the most of them and no replacement characters. A smaller alternative: when --translate-from is ja or zh and the input is not UTF-8, use the matching code page as the fallback encoding.

This report was drafted with the help of an AI assistant. The reproduction steps, outputs and source references were verified against seconv 5.4.0-beta6 and the linked lines before posting.

Dominant language
C#
Stars
14.8k
Forks
1.4k
Avg merge
1h 4m
Merged PRs (30d)
643

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from SubtitleEdit/subtitleedit

All issues in SubtitleEdit/subtitleedit

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.