Dynamic and Discriminated Unions

Comece lendo as propostas de DynamicUnion e DiscriminatedUnion e, em seguida, acompanhe o rascunho referenciado do PEP de Sealed Typing, o PEP 622 e pyright#5933. O trabalho estará concluído quando houver um design de typing acordado que abranja o momento da criação de subclasses, a validação do discriminador e o comportamento do type-checker; a issue não nomeia arquivos de implementação nem testes.

When modeling semi-structured data, discriminated unions are an invaluable tool in your arsenal.
It allows you to model a variable unit of data, where each unit concisely specifies its type and payload. This is a common pattern in event-driven architectures, where you have a stream of "events", "commands", or "actions", each of which is a discriminated union of event types.

I'd like to propose two new features to the typing module that formalize dynamic and discriminated unions.

1️⃣ Dynamic Unions

Current state

Assume we have a Status model that is a discriminated union of StatusA and StatusB:

from typing import Literal, Union
from pydantic import BaseModel


class StatusBase(BaseModel):
    status: str


class StatusA(StatusBase):
    status: Literal['a']


class StatusB(StatusBase):
    status: Literal['b']

To define a union that is understood by the type-checker at static-analysis time, you declare it by mentioning every member of the union:

Status = Union[StatusA, StatusB]
reveal_type(Status)  # Type of "Status" is "type[StatusA] | type[StatusB]"
print(Status)  # typing.Union[__main__.StatusA, __main__.StatusB]

Whenever adding a new subclass of StatusBase, this declaration must be updated.

I often abuse a pattern of lazily declaring a dynamic union instead of maintaining a static one, by tying all the members of the union to the subclasses of a parent class (in this case StatusBase):

Status = Union[tuple(StatusBase.__subclasses__())]
reveal_type(Status)  # Type of "Status" is "Unknown"
print(Status)  # typing.Union[__main__.StatusA, __main__.StatusB]

The type-checker is not able to infer the type of Status, because it's only compiled at runtime.

Proposed feature

I propose a new feature that would allow the type-checker to infer a dynamic union of all StatusBase subclasses, and finally error if more subclasses are declared afterward.

from typing import DynamicUnion  # Currently not a thing

Status = DynamicUnion[StatusBase]
reveal_type(Status)  # Type of "Status" is "type[StatusA] | type[StatusB]"
print(Status)  # typing.Union[__main__.StatusA, __main__.StatusB]

(Maybe call it SubclassUnion instead of DynamicUnion?)

The type-checker should detect if a subclass of StatusBase is declared after Status, and raise an error.

Status = DynamicUnion[StatusBase]

class StatusC(StatusBase):  # Error: Subclass of StatusBase declared after declaration of DynamicUnion[StatusBase]
    status: Literal['c']

This could work like the @sealed decorator was proposed to work in the Sealed Typing PEP draft. More useful information found on this email chain.

The @sealed decorator was designed as part of PEP 622 (structural pattern matching). Think of it as confining a type and its subclasses to a single module, so that the type-checker can dynamically infer a union of the subclasses. These are also known as algebraic data types.

I'd like to also explore the possibility of scoping algebraic data types to a package, or a dynamic scope bound between the parent class declaration and the union declaration.

2️⃣ Discriminated unions

Current state

For lack of a formalization of discriminated unions in core python, data (de)serialization libraries turn to their own objects. For example, pydantic specifies a discriminator value with Field, and uses Annotated to tag the union:

from typing import Annotated
from pydantic import Field

AnnotatedStatus = Annotated[
    Status,
    Field(discriminator='status')
]

This does not raise an error at runtime if any member of the Status union does not have a status field UNTIL AnnotatedStatus is used as a type on the field of a pydantic model declaration.

Annotated is an "escape hatch" that allows clients to express things that type-checkers
might not understand.

However, there is a push by third-party libraries for objects in the typing module that would canonically be used with the Annotated type. It signals a willingness to move from redundant proprietary internals toward ubiquitous core python. See PEP 727 and related discussion.

Proposed feature

I propose a new feature that would allow the type-checker to infer the type of AnnotatedStatus, by declaring it as a dynamic discriminated union:

from typing import DiscriminatedUnion  # Currently not a thing

AnnotatedStatus = DiscriminatedUnion[Status, 'status']
reveal_type(AnnotatedStatus)  # Type of "AnnotatedStatus" is "type[StatusA] | type[StatusB]"
print(AnnotatedStatus)  # typing.Union[__main__.StatusA, __main__.StatusB]

This would grant data (de)serialization libraries unified plumbing to declare discriminated unions, and would allow the type-checker to throw an error at static-analysis time if any member of the Status union does not have a status field, or if any member of the Status union has a status field with a value that is not unique among all members of the union.

The latter hinges on type-checkers being able to implicitly detect disjoint unions. See pyright#5933 for more information on mutable field invariance and implicit disjoint unions.

Motivation

When I'm writing JSONSchemas or deserializing in event-driven architectures, and I'm iterating fast, I gravitate toward easily being able to add new members to a union.

I usually do this in one of two ways. Either:

• all in one file, declaring parent, subclasses, then the union (as shown in the example above), or
• all in one directory, declaring the parent in base.py, each of the subclasses in a separate file, and the union in the module’s __init__.py

While this approach feels more magical than maintaining a static union yourself, it facilitates an accessible plug-in architecture, where users can implement new subclasses of the parent class in a new file, and have it automatically become part of the union without deeply understanding the internals of the library.