ImageGen generate-batch aborts valid JSONL jobs containing a Unicode line separator in poster text

Open Beginner friendly
  • Is something wrong?
#61,442 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
78/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
tooling

Research direction

The bug lives in the batch loader of codex-rs/skills/src/assets/samples/imagegen/scripts/image_gen.py, around line 464, where the whole UTF-8 input is split with str.splitlines(), which breaks on U+2028 inside JSON strings. Reproduce with the dry-run commands in the issue using a literal U+2028 versus its \u2028 escape. Done when the literal-character file lists the same two jobs as the escaped one, with the poster text preserved verbatim.

Written by the indexing model from the issue text.

Description

bug imagen skills

One literal U+2028 LINE SEPARATOR in a poster's text makes generate-batch reject the entire UTF-8 JSONL input before processing any jobs. Escaping the same character as \u2028 allows both jobs to proceed with identical decoded text. This affects producers that emit UTF-8 directly, including Python's json.dumps(..., ensure_ascii=False).

Steps to reproduce From the Codex checkout at de83e7674a69b9ae18544dd1fbf53f31bfe8c640, with Python 3.12.14. Dry-run mode requires no credentials.

d=$(mktemp -d)
python3 - "$d" <<'PY'
import json, sys
from pathlib import Path
jobs = [
    {"prompt": "A welcome poster", "text": "OPEN HOUSE\u2028SATURDAY"},
    {"prompt": "A second poster"},
]
for filename, escape in [("literal.jsonl", False), ("escaped.jsonl", True)]:
    path = Path(sys.argv[1]) / filename
    with path.open("w", encoding="utf-8") as f:
        for job in jobs:
            print(json.dumps(job, ensure_ascii=escape), file=f)
    with path.open(encoding="utf-8") as f:
        assert list(map(json.loads, f)) == jobs
PY
script=codex-rs/skills/src/assets/samples/imagegen/scripts/image_gen.py
python3 "$script" generate-batch --dry-run --input "$d/literal.jsonl" --out-dir "$d/out1"
python3 "$script" generate-batch --dry-run --input "$d/escaped.jsonl" --out-dir "$d/out2"

Observed The first invocation exits 1 with Invalid JSON on line 1: Unterminated string starting at: line 1 column 40 (char 39). The second lists jobs 1 and 2. The independent parser check passes for both files.

Expected Both batches should process the same two jobs, preserving the verbatim poster text.

Suggested fix Iterate over the UTF-8 file by physical lines instead of calling str.splitlines() on the whole document. The latter treats U+2028 inside a JSON string as a record boundary. Batch loader

Dominant language
Rust
Stars
135k
Forks
21.4k
Avg merge
1m
Merged PRs (30d)
997

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from openai/codex

All issues in openai/codex

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.