UnicodeDecodeError when decoding GeoDiff stderr on Windows hides the original error
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 68/100
Research direction
Start in the _run_geodiff() function at the res.stdout.decode() and res.stderr.decode() calls. Reproduce the Windows failure with non-UTF-8 GeoDiff output, then verify that the original GeoDiff error remains visible and decoding no longer raises UnicodeDecodeError.
Written by the indexing model from the issue text.
Description
Environment
• Mergin DB Sync 2.3.0
• GeoDiff 2.3.0
• Windows 11
• Python 3.12
Description
While investigating another issue, I found that Mergin DB Sync crashes with a UnicodeDecodeError when GeoDiff returns an error message containing non-UTF-8 characters on Windows.
Instead of reporting the original GeoDiff error, DB Sync terminates while decoding the stderr output produced by GeoDiff.
The relevant code is:
geodiff_stderr = res.stderr.decode()
This assumes that the error output is UTF-8 encoded. On Windows, this assumption may not hold depending on how GeoDiff emits its error messages.
Actual Result
DB Sync terminates with:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x97 in position ...
As a consequence, the original GeoDiff error is hidden, making it difficult to diagnose the underlying problem.
Investigation
To investigate the issue, I modified the run_geodiff() function to preserve the original stdout and stderr output.
Instead of:
geodiff_stderr = res.stderr.decode()
I temporarily changed the code to:
geodiff_stdout = res.stdout.decode(errors="replace")
geodiff_stderr = res.stderr.decode(errors="replace")
This allowed DB Sync to continue running and display the original GeoDiff error.
In my case, the hidden error was:
Error: Missing 'modified' file when opening sqlite driver:
C:/Users/.../camada_mergin_maps—_estacoes_amostragem_teste__estaes_de_amostragem.gpkg
Without this modification, the actual GeoDiff error could not be identified because DB Sync terminated first with the UnicodeDecodeError.
Expected Result
DB Sync should always display the original GeoDiff error, regardless of the encoding used by the underlying process.
It should not terminate while decoding the stderr output.
Possible Improvement
Instead of:
geodiff_stderr = res.stderr.decode()
consider one of the following approaches:
geodiff_stderr = res.stderr.decode(errors="replace")
or
import locale
geodiff_stderr = res.stderr.decode(
locale.getpreferredencoding(False),
errors="replace",
)
Either approach would prevent DB Sync from crashing while still preserving the original GeoDiff error message.
Related Issue
While investigating this behaviour, I identified the underlying GeoDiff issue:
MerginMaps/geodiff#258
DB Sync currently hides that error because of the UTF-8 decoding failure.
I'd be happy to test a fix on Windows if needed.
- Dominant language
- Python
- Stars
- 53
- Forks
- 24
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from MerginMaps/db-sync
-
Difficulty 3/5 1-2 days Newbie friendliness 38/100
MerginMaps/db-sync#185 · 1 comment ·
-
db-sync enters an infinite drop/recreate loop when geodiff init fails on invalid source geometries Open
Difficulty 4/5 3-5 days Newbie friendliness 42/100
MerginMaps/db-sync#182 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 52/100
MerginMaps/db-sync#181 · 2 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
MerginMaps/db-sync#179 · 3 comments ·
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
MerginMaps/db-sync#178 ·
All issues in MerginMaps/db-sync
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
syfoud/Simulated_Scepter#172 ·
-
A cancelled tests run makes the coverage comment workflow fail and reports it as a red check on main Openarea: ci bug perceived difficulty: 3
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Nitjsefnie-Harness-Commons/daedalus#921 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
EleutherAI/lm-evaluation-harness#4207 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
ClickHouse/clickhouse-connect#1057 ·