hound-search/hound

Latin charset=ISO-8859-1 parse

Open

#368 建立於 2020年11月3日

在 GitHub 查看
 (2 留言) (0 反應) (0 負責人)JavaScript (5,470 star) (606 fork)batch import
enhancementhelp wanted

描述

Hi there!

We have localized git, which store translations to languages , therefore ÖÄÕÜ etc ISO-8859-1 compliant files seem to be excluded as "Invalid UTF-8". How to overcome this problem, unfortunately i have no experience with GO to help with PR right now. Help is appreciated. for example: Lähme põõsasse sööma

貢獻者指南

技術棧
go
領域
clisearch
議題類型
bug
難度面向新貢獻者的預計實作難度，1 表示很小改動，5 表示專家級工作。
3
預計時間有經驗貢獻者完成調查、實作、測試並準備 pull request 的粗略時間範圍。
half day
活動狀態議題目前的可參與程度：新鮮、活躍、陳舊、阻塞或等待維護者輸入。
stale
清晰度議題是否清楚說明預期改動、驗收標準和下一步。
mostly clear
前置要求
basic understanding of Gocharacter encoding fundamentals
新手友善度1-100 的估計分數，表示該議題對首次貢獻者的友善程度。
40
研究方向
Investigate how Hound currently reads and validates file content for UTF 8 in the indexing process. Look at the file reading functions in the Go codebase, likely in the 'index' or 'search' package. Consider using the golang.org/x/text/encoding package to detect and convert non UTF 8 encodings such as ISO 8859-1. Check if there are any existing configuration options for encoding. Implement a fallback or detection mechanism to allow Latin 1 encoded files to be indexed without being rejected.