[BUG] Agent.chat(output_pydantic=Model) returns raw str — structured output contract broken (PAI-SDK-010)

Aperta
#5,590 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

  • #5593 di @praisonai-triage-agent — chiusa senza merge

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
45/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
ai

Direzione di ricerca

Start in praisonaiagents/agent/chat_mixin.py, where the output_pydantic branch returns the raw response text in both the sync and async blocks, and where use_native_format strips the prompt-side JSON instruction from build_messages. Reproduce with a small Pydantic model and a live or mocked gpt-4o-mini response that returns prose. Done means chat() with output_pydantic returns a validated model instance or raises a typed error, with a test covering both blocks. The return-type choice is still open in the thread, so confirm it with maintainers first.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug claude documentation

[BUG] Agent.chat(output_pydantic=Model) never returns a Pydantic instance — structured-output contract is a no-op on the live OpenAI path

Metadata

Field Value
Repository https://github.com/MervinPraison/PraisonAI
Package praisonaiagents
Version tested praisonaiagents 1.7.9 on release tag v4.7.11 (080796472) — also reproduced on 1.7.0 / 4.7.0
Wrapper praisonai 4.7.11
Labels bug, structured-output, pydantic, Agent.chat, sdk-contract, core-sdk
Severity High — public API advertises typed structured output; live call returns unstructured str and does not fail
Component src/praisonai-agents/praisonaiagents/agent/chat_mixin.py
Discovered Live OpenAI E2E (gpt-4o-mini), re-confirmed on v4.7.11
Related (not duplicate) #3657 hierarchical schema (fixed); #5463 completion model override vs constructor model; #3703 Task output_pydantic path (closed — different entrypoint)
Tracking PAI-SDK-010

Executive summary

Agent.chat(..., output_pydantic=SomeModel) is the SDK's typed-output surface. Callers reasonably expect either:

  1. a validated SomeModel instance, or
  2. a hard error if the model cannot satisfy the schema.

Live E2E on praisonaiagents==1.7.0 with OpenAI gpt-4o-mini did neither. The call returned a plain English string:

'The capital of France is Paris.'

isinstance(out, Pair) was False. type(out) was str. No exception. Exit path looks successful.

That is a silent contract violation. Downstream code that does result.city will raise AttributeError in production. Downstream code that does json.loads(result) will also fail because the payload is prose, not JSON.

Root cause is in chat_mixin.py: even after building a native response_format JSON schema, the success path returns the raw response_text string and never instantiates the Pydantic model. The comment in source is literal: "Add to chat history and return raw response".


Labels

  • bug
  • structured-output

Live environment

Field Value
OS Windows 10/11
Python 3.13
praisonaiagents 1.7.9 (tag v4.7.11)
praisonai 4.7.11
Model gpt-4o-mini via LiteLLM / OpenAI
Date 2026-10-02 (v4.7.11 validation); originally 2026-08-25
Git ref v4.7.11 @ 08079647275c500f27e02ce7fa229f15b0308f8f

Step-by-step reproduction (live)

from pydantic import BaseModel, Field
from praisonaiagents import Agent

class Pair(BaseModel):
    city: str = Field(...)
    country: str = Field(...)

agent = Agent(instructions="Return structured data only.", llm="gpt-4o-mini")
out = agent.chat("Capital of France.", output_pydantic=Pair)

print(type(out).__name__)
print(repr(out))
print("is_model", isinstance(out, Pair))
Expected
Check Expected
Return type Pair
out.city "Paris"
out.country "France"
Exception none, or a typed validation error if the model cannot comply
Actual (live, 2.2s, HTTP 200 from the provider)
Check Actual
Return type str
repr(out) 'The capital of France is Paris.'
isinstance(out, Pair) False
Exception none

Live E2E record (v4.7.11, case structured_pydantic, ~24s):

{
  "type": "str",
  "repr": "'The capital of France is Paris.'",
  "is_model": false,
  "is_str": true,
  "contract_ok": false
}

Strict manual E2E fails with AssertionError: expected Pair model, got str. Integration tests that only assert “no exception” will false-green while the SDK contract is broken.


Why this makes the SDK weak

Consumer What they wrote What they got
Typed pipeline city = out.city AttributeError: 'str' object has no attribute 'city'
JSON API gateway json.loads(out) JSONDecodeError (prose, not JSON)
Eval / golden tests assert out.country == "France" always fail, flaky if they str()-compare
Multi-agent handoff pass out as Task context specialist receives a sentence, not fields
README "Structured Output Agents" Pydantic model as the API undocumented string return

This is not a docs nit. Structured output is how an SDK becomes composable. Without it, every caller re-implements JSON parsing and retry.


Architectural context

┌──────────────────────────────────────────────────────────────────────────┐
│  Caller:  agent.chat(prompt, output_pydantic=Pair)                       │
└──────────────────────────────────────────────────────────────────────────┘
                    │
                    ▼
┌──────────────────────────────────────────────────────────────────────────┐
│  ChatMixin._chat_impl                                                    │
│    1. schema_model = Pair                                                │
│    2. if _supports_native_structured_output():                           │
│         response_format = _build_response_format(Pair)   # json_schema   │
│         use_native_format = True                                         │
│    3. _build_messages(..., use_native_format=True)                       │
│         → SKIPS prompt-side "respond with JSON matching schema"          │
│    4. _chat_completion(..., response_format=response_format)             │
│    5. response_text = message.content.strip()                            │
│    6. if output_pydantic:                                                │
│         return raw response_text          ← CONTRACT BREAKS HERE         │
│         # never Pair.model_validate_json(response_text)                  │
└──────────────────────────────────────────────────────────────────────────┘
                    │
                    ▼
┌─────────────┐     ┌──────────────┐     ┌─────────────────────┐
│  Pair       │     │  JSON str    │     │  English prose      │
│  (promised) │     │  (minimum)   │     │  (what we got)      │
└─────────────┘     └──────────────┘     └─────────────────────┘

Sequence:

sequenceDiagram
    participant C as Caller
    participant Chat as ChatMixin
    participant Cap as supports_structured_outputs
    participant LLM as LiteLLM/OpenAI

    C->>Chat: chat("Capital of France.", output_pydantic=Pair)
    Chat->>Cap: _supports_native_structured_output()
    Cap-->>Chat: True (gpt-4o-mini)
    Note over Chat: use_native_format=True<br/>prompt JSON instruction SKIPPED
    Chat->>LLM: _chat_completion(response_format=json_schema Pair)
    LLM-->>Chat: content = "The capital of France is Paris."
    Note over Chat: no Pair.model_validate_json()
    Chat-->>C: str  (not Pair, not JSON)

Current architecture (code evidence)

1. Native format is computed, then ignored at return

chat_mixin.py (installed 1.7.0; same comment still on origin/main):

response = self._chat_completion(
    messages, temperature=temperature, tools=tools, ...,
    response_format=response_format,
)
content = response.choices[0].message.content
response_text = content.strip() if content else ""

# Handle output_json or output_pydantic if specified
if output_json or output_pydantic:
    # Add to chat history and return raw response
    ...
    return self._trigger_after_agent_hook(original_prompt, validated_response, start_time)

validated_response is the guardrail result of a string. There is no schema_model.model_validate_json(...).

2. When native format is on, the fallback JSON instruction is stripped
messages, original_prompt = self._openai_client.build_messages(
    ...
    output_json=None if use_native_format else output_json,
    output_pydantic=None if use_native_format else output_pydantic
)

So if LiteLLM/OpenAI does not actually constrain the completion (provider quirk, dropped response_format, or parsed field unused), the model is not even asked in the prompt to emit JSON. That matches the live prose answer.

3. chat() is typed as Optional[str]
def chat(self, prompt: str, ..., output_pydantic: Optional[Any] = None, ...) -> Optional[str]:

The type hint documents the bug. A parameter named output_pydantic whose return type cannot be a Pydantic model is an API lie.

4. Hierarchical manager was already taught the lesson

process/manager_schema.py in 1.7.0 uses ConfigDict(extra="forbid") and parses into ManagerInstructions. User-level output_pydantic did not get the same treatment. The SDK is inconsistent with itself.


Expected vs unexpected (table)

Step Expected Unexpected (live)
Build response_format OpenAI-strict JSON schema for Pair Happens internally; unused at return
Prompt Schema in response_format or explicit JSON instruction Native on → instruction stripped
LLM content {"city":"Paris","country":"France"} "The capital of France is Paris."
Parse Pair.model_validate_json(...) skipped
Return Pair str
Failure mode ValidationError / retry silent success
Tests assert type + fields none covering this live path

Impact analysis

Stakeholder Impact
SDK users Typed pipelines crash one hop later
Eval authors False-green if they only check bool(result)
AgentFlow / teams Cannot pass structured fields between steps without regex
OpenAI spend Extra retries when callers wrap their own JSON repair
Trust README "Structured Output Agents" is not the live chat() path
Severity justification
Factor Assessment
Functional correctness Broken contract — wrong type, wrong shape
User visibility Low — no exception
Fix complexity Moderate — parse + retry already exist for manager / guardrails
Scope Every output_pydantic / output_json chat()/achat() caller

Recommended architecture

if schema_model:
    response_text = ...
    try:
        data = json.loads(response_text)
        parsed = schema_model.model_validate(data)
    except (json.JSONDecodeError, ValidationError) as exc:
        # retry with explicit JSON instruction (existing guardrail retry pattern)
        raise StructuredOutputError(...) from exc
    return parsed

Also:

  1. Use response.choices[0].message.parsed when the OpenAI parse/beta path is used (already done for ReflectionOutput).
  2. Keep prompt JSON instruction if native response_format is None or LiteLLM dropped it.
  3. Change return type to Union[str, BaseModel, None] or a small ChatResult with .text and .parsed.
  4. Fail loud after N retries. Never return prose when output_pydantic was requested.
Minimal code fix (chat_mixin.py, both sync and async blocks)
if output_json or output_pydantic:
    schema_model = output_pydantic or output_json
    parsed = None
    if hasattr(schema_model, "model_validate_json") and response_text:
        try:
            parsed = schema_model.model_validate_json(response_text)
        except Exception:
            parsed = None
    if parsed is None and hasattr(schema_model, "model_validate"):
        try:
            parsed = schema_model.model_validate(json.loads(response_text))
        except Exception:
            parsed = None
    if parsed is None:
        raise ValueError(
            f"Agent {self.name}: output_pydantic={schema_model.__name__} "
            f"was requested but the model returned unparseable content: "
            f"{response_text[:200]!r}"
        )
    self._append_to_chat_history({"role": "assistant", "content": response_text})
    self._persist_message("assistant", response_text)
    return parsed

Wire the same into _achat_impl (there is a second copy of the "return raw response" comment on origin/main).


Testing strategy

Live E2E (must hit a real provider — this bug is invisible under mocks that return JSON strings the caller never type-checks):

def test_chat_output_pydantic_returns_model():
    class Pair(BaseModel):
        city: str
        country: str
    agent = Agent(instructions="Structured only.", llm="gpt-4o-mini")
    out = agent.chat("Capital of France.", output_pydantic=Pair)
    assert isinstance(out, Pair)
    assert out.city.lower() == "paris"
    assert "france" in out.country.lower()

Unit:

  • Mock _chat_completion returning prose → must raise, not return str.
  • Mock returning valid JSON → must return Pair.
  • use_native_format=True with empty parsed field still parses content.

Acceptance criteria

  • agent.chat(..., output_pydantic=M) returns an instance of M on success.
  • Invalid content raises a dedicated error after retries (not a silent string).
  • Sync and async paths behave the same.
  • Live test against OpenAI gpt-4o-mini is in CI or a documented nightly E2E.
  • chat() type hints match the contract.
  • README structured-output example uses this API and is executed in CI.

Workaround (today)

Do not trust output_pydantic on chat(). Parse yourself:

import json
from pydantic import BaseModel

raw = agent.chat("Capital of France. Reply JSON: {\"city\",\"country\"}.")
# still may be prose — add your own retry

Or use Task(output_pydantic=...) if that path still parses (verify separately — do not assume).


Maintainer notes

  • Do not close as "works with Task.output_pydantic" without a live Agent.chat test. The advertised SDK surface is chat/start.
  • Do not treat this as docs-only. The implementation comment says "return raw response".
  • Related (do not conflate): PAI-SDK-009 hierarchical schema — already fixed in 1.7.0; live hierarchical E2E on 2026-08-25 returned "4" correctly.

SDK improvement value

Fixing this unlocks typed agent pipelines (evals, multi-step handoffs, API gateways) without every consumer re-implementing JSON repair. Aligns Agent.chat() with hierarchical manager parsing already shipped in 1.7.x.

Verification commands

git checkout v4.7.11
pip install -e src/praisonai-agents -e src/praisonai
export OPENAI_API_KEY=...
python -c "
from pydantic import BaseModel, Field
from praisonaiagents import Agent
class Pair(BaseModel):
    city: str = Field(...)
    country: str = Field(...)
out = Agent(instructions='Structured only.', llm='gpt-4o-mini').chat('Capital of France.', output_pydantic=Pair)
assert isinstance(out, Pair), (type(out), out)
"
Lingua principale
Python
Stelle
9.3k
Fork
1.5k
Merge medio
1g 5h
PR unite (30g)
520

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di MervinPraison/PraisonAI

Tutte le issue di MervinPraison/PraisonAI

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.