[API Proposal]: Allow Utf8JsonReader to be driven by a custom token source

Abierto
  • ¿Algo no está bien?
#136,620 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
25/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
csharp
Área
api, backend

Línea de trabajo

Start by reading Utf8JsonReader and JsonSerializer implementation and tests in the .NET runtime, focusing on reader copies, look-ahead, and token value semantics. The proposal names no specific files or tests, so first locate the existing reader and serializer test suites and trace their current byte-based paths. Done means the proposed source-backed reader and synchronous deserialization APIs work with converters and generated metadata, preserve source positions, and avoid measurable regression to the existing read path.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area-System.Text.Json untriaged
Background and motivation

System.Text.Json has a mature serializer, metadata model, source generator and converter ecosystem. Many JSON-adjacent formats (JSON5, HJSON, JSONC variants, and anything that can produce a JSON-shaped token stream) would like to reuse all of that, but the only entry point is Utf8JsonReader, which can only tokenize strict JSON bytes.

JsonConverter<T>.Read takes a concrete ref struct Utf8JsonReader, so there is no way to supply tokens from another tokenizer. Today the only option is to transcode the input to JSON bytes and deserialize those. For example, Json5 tokenizes JSON5, writes it through Utf8JsonWriter into a buffer, and then calls JsonSerializer.Deserialize<T>. This works, but:

  • The whole document (or every chunk) has to be re-encoded and copied.
  • JsonException line/position information refers to the transcoded JSON, not to the user's source file. For config-file formats this is the main usability problem.
  • Every library has to rebuild the transcoding and buffering plumbing itself.

This proposal does not ask for System.Text.Json to support JSON5 or any other format. It asks for a format-neutral extension point: let a Utf8JsonReader pull tokens from a caller-supplied source that has already normalized each value to JSON.

API Proposal
namespace System.Text.Json;

public abstract class JsonTokenSource
{
    /// Advances to the next token. Returns false at end of input.
    /// `value` must be a normalized JSON representation of the token's value
    /// (JSON string escaping, JSON number syntax), valid until the next call to Read.
    public abstract bool Read(out JsonTokenType tokenType, out ReadOnlySpan<byte> value, out bool valueIsEscaped);

    /// Position of the current token in the original input, used for TokenStartIndex and exception messages.
    public abstract long TokenStartIndex { get; }
    public virtual long LineNumber => -1;
    public virtual long BytePositionInLine => -1;

    /// Supports reader copies used for look-ahead (e.g. polymorphic metadata, read-ahead in the serializer).
    public abstract object? Checkpoint();
    public abstract void Restore(object? checkpoint);
}

public ref partial struct Utf8JsonReader
{
    public Utf8JsonReader(JsonTokenSource source, JsonReaderOptions options = default);
}

public static partial class JsonSerializer
{
    public static TValue? Deserialize<TValue>(JsonTokenSource source, JsonSerializerOptions? options = null);
    public static TValue? Deserialize<TValue>(JsonTokenSource source, JsonTypeInfo<TValue> jsonTypeInfo);
    public static object? Deserialize(JsonTokenSource source, Type returnType, JsonSerializerOptions? options = null);
}

When constructed with a source, Read() delegates to it, while depth tracking, structural validation, Skip(), TrySkip(), GetString(), TryGet*(), ValueTextEquals() and so on continue to work over the normalized values.

API Usage
public sealed class Json5TokenSource(ReadOnlyMemory<byte> json5) : JsonTokenSource
{
    // Tokenizes JSON5; emits 'it\'s' as "it's", 0x1F as 31, unquoted keys as PropertyName, etc.
}

var config = JsonSerializer.Deserialize<AppConfig>(
    new Json5TokenSource(File.ReadAllBytes("appsettings.json5")),
    options); // all custom converters and source-generated metadata work unchanged

// Errors report the position in appsettings.json5:
// "'x' is an invalid start of a value. LineNumber: 12 | BytePositionInLine: 4."
Alternative Designs
  • Interface instead of abstract class (IJsonTokenSource). An abstract class leaves room to add members later (e.g. ValueSequence support) without breaking implementers.
  • Exception position remapping only: a hook on JsonSerializerOptions that rewrites LineNumber/BytePositionInLine in a JsonException. This is much smaller and fixes the biggest usability problem for transcoding libraries, but still requires the copy.
  • Status quo: transcode to JSON and deserialize. This works, but has the costs described above.
Risks
  • Utf8JsonReader.Read() is a very hot path. The extra branch must not measurably regress the existing byte-based path. Keeping the source-backed path out of line behind a single null/flag check should keep it close to free.
  • Reader copies: the serializer copies the reader struct for look-ahead and restore. A stateful source has to support that, hence Checkpoint/Restore. Code that assumes a copied reader is fully independent needs auditing.
  • ValueSpan, HasValueSequence and ValueIsEscaped semantics: the source must guarantee that values are valid JSON tokens. Invalid normalized values should fail with the same errors as invalid JSON.
  • Async/streaming deserialization (DeserializeAsync, resumable JsonReaderState) is out of scope for the first version. Sync-only overloads are proposed.
  • JsonDocument/JsonNode parsing from a source is out of scope here and could be added later.
Lenguaje dominante
C#
Estrellas
18.4k
Forks
5.8k
Merge medio
2 d 12 h
PR fusionados (30 d)
600

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de dotnet/runtime

Todos los issues de dotnet/runtime

Issues similares

Más issues de C#

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.