[API Proposal]: Allow Utf8JsonReader to be driven by a custom token source

Start by reading Utf8JsonReader and JsonSerializer implementation and tests in the .NET runtime, focusing on reader copies, look-ahead, and token value semantics. The proposal names no specific files or tests, so first locate the existing reader and serializer test suites and trace their current byte-based paths. Done means the proposed source-backed reader and synchronous deserialization APIs work with converters and generated metadata, preserve source positions, and avoid measurable regression to the existing read path.

Background and motivation

System.Text.Json has a mature serializer, metadata model, source generator and converter ecosystem. Many JSON-adjacent formats (JSON5, HJSON, JSONC variants, and anything that can produce a JSON-shaped token stream) would like to reuse all of that, but the only entry point is Utf8JsonReader, which can only tokenize strict JSON bytes.

JsonConverter<T>.Read takes a concrete ref struct Utf8JsonReader, so there is no way to supply tokens from another tokenizer. Today the only option is to transcode the input to JSON bytes and deserialize those. For example, Json5 tokenizes JSON5, writes it through Utf8JsonWriter into a buffer, and then calls JsonSerializer.Deserialize<T>. This works, but:

• The whole document (or every chunk) has to be re-encoded and copied.
• JsonException line/position information refers to the transcoded JSON, not to the user's source file. For config-file formats this is the main usability problem.
• Every library has to rebuild the transcoding and buffering plumbing itself.

This proposal does not ask for System.Text.Json to support JSON5 or any other format. It asks for a format-neutral extension point: let a Utf8JsonReader pull tokens from a caller-supplied source that has already normalized each value to JSON.

API Proposal

namespace System.Text.Json;

public abstract class JsonTokenSource
{
    /// Advances to the next token. Returns false at end of input.
    /// `value` must be a normalized JSON representation of the token's value
    /// (JSON string escaping, JSON number syntax), valid until the next call to Read.
    public abstract bool Read(out JsonTokenType tokenType, out ReadOnlySpan<byte> value, out bool valueIsEscaped);

    /// Position of the current token in the original input, used for TokenStartIndex and exception messages.
    public abstract long TokenStartIndex { get; }
    public virtual long LineNumber => -1;
    public virtual long BytePositionInLine => -1;

    /// Supports reader copies used for look-ahead (e.g. polymorphic metadata, read-ahead in the serializer).
    public abstract object? Checkpoint();
    public abstract void Restore(object? checkpoint);
}

public ref partial struct Utf8JsonReader
{
    public Utf8JsonReader(JsonTokenSource source, JsonReaderOptions options = default);
}

public static partial class JsonSerializer
{
    public static TValue? Deserialize<TValue>(JsonTokenSource source, JsonSerializerOptions? options = null);
    public static TValue? Deserialize<TValue>(JsonTokenSource source, JsonTypeInfo<TValue> jsonTypeInfo);
    public static object? Deserialize(JsonTokenSource source, Type returnType, JsonSerializerOptions? options = null);
}

When constructed with a source, Read() delegates to it, while depth tracking, structural validation, Skip(), TrySkip(), GetString(), TryGet*(), ValueTextEquals() and so on continue to work over the normalized values.

API Usage

public sealed class Json5TokenSource(ReadOnlyMemory<byte> json5) : JsonTokenSource
{
    // Tokenizes JSON5; emits 'it\'s' as "it's", 0x1F as 31, unquoted keys as PropertyName, etc.
}

var config = JsonSerializer.Deserialize<AppConfig>(
    new Json5TokenSource(File.ReadAllBytes("appsettings.json5")),
    options); // all custom converters and source-generated metadata work unchanged

// Errors report the position in appsettings.json5:
// "'x' is an invalid start of a value. LineNumber: 12 | BytePositionInLine: 4."

Alternative Designs

• Interface instead of abstract class (IJsonTokenSource). An abstract class leaves room to add members later (e.g. ValueSequence support) without breaking implementers.
• Exception position remapping only: a hook on JsonSerializerOptions that rewrites LineNumber/BytePositionInLine in a JsonException. This is much smaller and fixes the biggest usability problem for transcoding libraries, but still requires the copy.
• Status quo: transcode to JSON and deserialize. This works, but has the costs described above.

Risks

• Utf8JsonReader.Read() is a very hot path. The extra branch must not measurably regress the existing byte-based path. Keeping the source-backed path out of line behind a single null/flag check should keep it close to free.
• Reader copies: the serializer copies the reader struct for look-ahead and restore. A stateful source has to support that, hence Checkpoint/Restore. Code that assumes a copied reader is fully independent needs auditing.
• ValueSpan, HasValueSequence and ValueIsEscaped semantics: the source must guarantee that values are valid JSON tokens. Invalid normalized values should fail with the same errors as invalid JSON.
• Async/streaming deserialization (DeserializeAsync, resumable JsonReaderState) is out of scope for the first version. Sync-only overloads are proposed.
• JsonDocument/JsonNode parsing from a source is out of scope here and could be added later.