Offset encoding in LSIF dumps

No se nombran archivos, pruebas ni puntos de entrada. Empieza leyendo el formato LSIF y las definiciones de metadatos; después, compara las tres opciones de codificación descritas aquí. El trabajo estaría terminado con un modelo de codificación decidido y los cambios correspondientes en el formato, pero este issue registra una discusión sin resolver en lugar de una tarea de implementación delimitada.

The LSP defines that character offsets in lines (e.g. the character property in Positions) is an index into a string encoded using uft-16. Strings are encoded in memory using utf-16 in programming languages like Java, JavaScript and .Net. However newer languages use utf-8 as the encoding for strings in memory. This causes some friction in the LSP mainly because:

• implementors of clients and servers are not aware of this
• there is no way right now to detect that the client and server are using a different encoding.

There is a push in LSP right now to improve this (no concrete action has been decided yet) but we should try to avoid the same problem in LSIF. So I would like to discuss possible options:

1. we pick one encoding that is supported in LSIF. This doesn't have to be utf-16. However to my knowledge there is no best pick. utf-32 would be great since it is a fixed encoding, but almost no programming language have support for it. So it requires conversion on all sides. utf-16 is bad for new programming lanaguges utf-8 for more traditional languages. Byte offsets would need to be defined on an encoding as well since we render characters in the UI not bytes. So no actual benefit.
2. we allow to create dumps with different encodings and store the choosen encoding in the meta data. This means that someone will need to convert if client and server don't speak the same encoding. The tendency in LSP is currently to make servers do the conversion since they usually have access to the files. A web interface for example would have a hard time doing the conversion for example for a find all reference result since it would need to fetch the content of all files mentioned in the references result. Note that this also requires that the file content is part of the dump.
3. we allow to store ranges with n encodings (pratically utf-8 and utf-16). This doesn't mean that every range is duplicated since as long as all characters on the line before the character offest are smaller than ASCII 127 the offsets are the same. So we could attach the converted ranges somehow to the orginial range.

I tend to go with a combination of 2. and 3. We store the ecodings with the dump. The first encoding is the primary encoding the ranges are in. All other encodings are scondary and the value can be reach from the primary range using an edge like utf-8 or utf-16