With --field-include-all-keys, a $ref value is written with an escape where the key beside it prints the same character

Open Beginner friendly
  • Is something wrong?
#6,906 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
74/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
tooling

Research direction

Start in python_literal.py at represent_python_value: find the branch that dispatches on str subclasses, which currently calls the interpreter's repr() and so leaks the running Python's Unicode table into the output. Read the legacy-str-subclass-host-repr entry in docs/architecture.md to understand why that branch exists before deciding whether ResolvedRef should go through stable_repr. Reproduce with the schema.json and command given, then check that both the keys and the $ref values print Unit꟎/Mass꟎ literally and that existing tests around the literal formatter still pass.

Written by the indexing model from the issue text.

Description

bug

With --field-include-all-keys, the generated json_schema_extra writes a definition name that holds a character the running Python does not yet print in two ways inside one literal: the key prints the letter, and the $ref value that points at it prints an escape. The two spell the same text, but the output now depends on the Python that runs the generator, and a newer Python writes the letter in both places.

The name below uses U+A7CE, a Latin letter added in Unicode 17. A Python with an older Unicode table (3.14 has Unicode 16) treats it as unprintable.

Reproduction

schema.json:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "definitions": {
    "Order": {"type": "object", "properties": {"unit": {"$ref": "#/definitions/Unit꟎"}}},
    "Unit꟎": {"type": "string", "description": "A unit", "allOf": [{"$ref": "#/definitions/Mass꟎"}]},
    "Mass꟎": {"type": "string", "enum": ["g", "kg"]}
  }
}

datamodel-codegen --input schema.json --input-file-type jsonschema --field-include-all-keys --disable-timestamp --formatters builtin

Output on Python 3.14 (the relevant lines):

            'definitions': {
                'Order': {
                    'type': 'object',
                    'properties': {'unit': {'$ref': '#/definitions/Unit꟎'}},
                },
                'Unit꟎': {
                    'type': 'string',
                    'description': 'A unit',
                    'allOf': [{'$ref': '#/definitions/Mass꟎'}],
                },
                'Mass꟎': {'type': 'string', 'enum': ['g', 'kg']},
            },

Expected

The same character is written the same way everywhere in the literal, and the output does not depend on the Python version that runs the generator: the $ref values are written as '#/definitions/Unit꟎' and '#/definitions/Mass꟎', like the keys beside them, as every other string of the generated module is written.

Since

0.89.0. In 0.86.0 to 0.88.0 both the keys and the values were written with the escape (consistent, but also host dependent); 0.89.0 to main write the keys with the letter and the $ref values with the escape.

Cause

python_literal.py:represent_python_value: the $ref values are a str subclass (ResolvedRef) and the case for a str subclass writes them with the running interpreter's repr(), while plain strings go through stable_repr, which writes characters as the newest supported target prints them. It is the kept difference legacy-str-subclass-host-repr in docs/architecture.md.

Dominant language
Python
Stars
4k
Forks
460
Avg merge
6h 12m
Merged PRs (30d)
923

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from datamodel-code-generator/datamodel-code-generator

All issues in datamodel-code-generator/datamodel-code-generator

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.