hound-search/hound

Latin charset=ISO-8859-1 parse

Open

#368 创建于 2020年11月3日

在 GitHub 查看
 (2 评论) (0 反应) (0 负责人)JavaScript (5,470 star) (606 fork)batch import
enhancementhelp wanted

描述

Hi there!

We have localized git, which store translations to languages , therefore ÖÄÕÜ etc ISO-8859-1 compliant files seem to be excluded as "Invalid UTF-8". How to overcome this problem, unfortunately i have no experience with GO to help with PR right now. Help is appreciated. for example: Lähme põõsasse sööma

贡献者指南

技术栈
go
领域
clisearch
议题类型
bug
难度面向新贡献者的预计实现难度，1 表示很小改动，5 表示专家级工作。
3
预计时间有经验贡献者完成调查、实现、测试并准备 pull request 的粗略时间范围。
half day
活动状态议题当前的可参与程度：新鲜、活跃、陈旧、阻塞或等待维护者输入。
stale
清晰度议题是否清楚说明期望改动、验收标准和下一步。
mostly clear
前置要求
basic understanding of Gocharacter encoding fundamentals
新手友好度1-100 的估计分数，表示该议题对首次贡献者的友好程度。
40
研究方向
Investigate how Hound currently reads and validates file content for UTF 8 in the indexing process. Look at the file reading functions in the Go codebase, likely in the 'index' or 'search' package. Consider using the golang.org/x/text/encoding package to detect and convert non UTF 8 encodings such as ISO 8859-1. Check if there are any existing configuration options for encoding. Implement a fallback or detection mechanism to allow Latin 1 encoded files to be indexed without being rejected.
Latin charset=ISO-8859-1 parse · hound-search/hound#368 | Good First Issue