[BUG] Agent.chat(output_pydantic=Model) returns raw str — structured output contract broken (PAI-SDK-010)
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
- #5593 di @praisonai-triage-agent — chiusa senza merge
Valutazione
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Idoneità per principianti
- 45/100
Direzione di ricerca
Start in praisonaiagents/agent/chat_mixin.py, where the output_pydantic branch returns the raw response text in both the sync and async blocks, and where use_native_format strips the prompt-side JSON instruction from build_messages. Reproduce with a small Pydantic model and a live or mocked gpt-4o-mini response that returns prose. Done means chat() with output_pydantic returns a validated model instance or raises a typed error, with a test covering both blocks. The return-type choice is still open in the thread, so confirm it with maintainers first.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
[BUG] Agent.chat(output_pydantic=Model) never returns a Pydantic instance — structured-output contract is a no-op on the live OpenAI path
Metadata
| Field | Value |
|---|---|
| Repository | https://github.com/MervinPraison/PraisonAI |
| Package | praisonaiagents |
| Version tested | praisonaiagents 1.7.9 on release tag v4.7.11 (080796472) — also reproduced on 1.7.0 / 4.7.0 |
| Wrapper | praisonai 4.7.11 |
| Labels | bug, structured-output, pydantic, Agent.chat, sdk-contract, core-sdk |
| Severity | High — public API advertises typed structured output; live call returns unstructured str and does not fail |
| Component | src/praisonai-agents/praisonaiagents/agent/chat_mixin.py |
| Discovered | Live OpenAI E2E (gpt-4o-mini), re-confirmed on v4.7.11 |
| Related (not duplicate) | #3657 hierarchical schema (fixed); #5463 completion model override vs constructor model; #3703 Task output_pydantic path (closed — different entrypoint) |
| Tracking | PAI-SDK-010 |
Executive summary
Agent.chat(..., output_pydantic=SomeModel) is the SDK's typed-output surface. Callers reasonably expect either:
- a validated
SomeModelinstance, or - a hard error if the model cannot satisfy the schema.
Live E2E on praisonaiagents==1.7.0 with OpenAI gpt-4o-mini did neither. The call returned a plain English string:
'The capital of France is Paris.'
isinstance(out, Pair) was False. type(out) was str. No exception. Exit path looks successful.
That is a silent contract violation. Downstream code that does result.city will raise AttributeError in production. Downstream code that does json.loads(result) will also fail because the payload is prose, not JSON.
Root cause is in chat_mixin.py: even after building a native response_format JSON schema, the success path returns the raw response_text string and never instantiates the Pydantic model. The comment in source is literal: "Add to chat history and return raw response".
Labels
bugstructured-output
Live environment
| Field | Value |
|---|---|
| OS | Windows 10/11 |
| Python | 3.13 |
praisonaiagents |
1.7.9 (tag v4.7.11) |
praisonai |
4.7.11 |
| Model | gpt-4o-mini via LiteLLM / OpenAI |
| Date | 2026-10-02 (v4.7.11 validation); originally 2026-08-25 |
| Git ref | v4.7.11 @ 08079647275c500f27e02ce7fa229f15b0308f8f |
Step-by-step reproduction (live)
from pydantic import BaseModel, Field
from praisonaiagents import Agent
class Pair(BaseModel):
city: str = Field(...)
country: str = Field(...)
agent = Agent(instructions="Return structured data only.", llm="gpt-4o-mini")
out = agent.chat("Capital of France.", output_pydantic=Pair)
print(type(out).__name__)
print(repr(out))
print("is_model", isinstance(out, Pair))
Expected
| Check | Expected |
|---|---|
| Return type | Pair |
out.city |
"Paris" |
out.country |
"France" |
| Exception | none, or a typed validation error if the model cannot comply |
Actual (live, 2.2s, HTTP 200 from the provider)
| Check | Actual |
|---|---|
| Return type | str |
repr(out) |
'The capital of France is Paris.' |
isinstance(out, Pair) |
False |
| Exception | none |
Live E2E record (v4.7.11, case structured_pydantic, ~24s):
{
"type": "str",
"repr": "'The capital of France is Paris.'",
"is_model": false,
"is_str": true,
"contract_ok": false
}
Strict manual E2E fails with AssertionError: expected Pair model, got str. Integration tests that only assert “no exception” will false-green while the SDK contract is broken.
Why this makes the SDK weak
| Consumer | What they wrote | What they got |
|---|---|---|
| Typed pipeline | city = out.city |
AttributeError: 'str' object has no attribute 'city' |
| JSON API gateway | json.loads(out) |
JSONDecodeError (prose, not JSON) |
| Eval / golden tests | assert out.country == "France" |
always fail, flaky if they str()-compare |
| Multi-agent handoff | pass out as Task context |
specialist receives a sentence, not fields |
| README "Structured Output Agents" | Pydantic model as the API | undocumented string return |
This is not a docs nit. Structured output is how an SDK becomes composable. Without it, every caller re-implements JSON parsing and retry.
Architectural context
┌──────────────────────────────────────────────────────────────────────────┐
│ Caller: agent.chat(prompt, output_pydantic=Pair) │
└──────────────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────┐
│ ChatMixin._chat_impl │
│ 1. schema_model = Pair │
│ 2. if _supports_native_structured_output(): │
│ response_format = _build_response_format(Pair) # json_schema │
│ use_native_format = True │
│ 3. _build_messages(..., use_native_format=True) │
│ → SKIPS prompt-side "respond with JSON matching schema" │
│ 4. _chat_completion(..., response_format=response_format) │
│ 5. response_text = message.content.strip() │
│ 6. if output_pydantic: │
│ return raw response_text ← CONTRACT BREAKS HERE │
│ # never Pair.model_validate_json(response_text) │
└──────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────┐ ┌──────────────┐ ┌─────────────────────┐
│ Pair │ │ JSON str │ │ English prose │
│ (promised) │ │ (minimum) │ │ (what we got) │
└─────────────┘ └──────────────┘ └─────────────────────┘
Sequence:
sequenceDiagram
participant C as Caller
participant Chat as ChatMixin
participant Cap as supports_structured_outputs
participant LLM as LiteLLM/OpenAI
C->>Chat: chat("Capital of France.", output_pydantic=Pair)
Chat->>Cap: _supports_native_structured_output()
Cap-->>Chat: True (gpt-4o-mini)
Note over Chat: use_native_format=True<br/>prompt JSON instruction SKIPPED
Chat->>LLM: _chat_completion(response_format=json_schema Pair)
LLM-->>Chat: content = "The capital of France is Paris."
Note over Chat: no Pair.model_validate_json()
Chat-->>C: str (not Pair, not JSON)
Current architecture (code evidence)
1. Native format is computed, then ignored at return
chat_mixin.py (installed 1.7.0; same comment still on origin/main):
response = self._chat_completion(
messages, temperature=temperature, tools=tools, ...,
response_format=response_format,
)
content = response.choices[0].message.content
response_text = content.strip() if content else ""
# Handle output_json or output_pydantic if specified
if output_json or output_pydantic:
# Add to chat history and return raw response
...
return self._trigger_after_agent_hook(original_prompt, validated_response, start_time)
validated_response is the guardrail result of a string. There is no schema_model.model_validate_json(...).
2. When native format is on, the fallback JSON instruction is stripped
messages, original_prompt = self._openai_client.build_messages(
...
output_json=None if use_native_format else output_json,
output_pydantic=None if use_native_format else output_pydantic
)
So if LiteLLM/OpenAI does not actually constrain the completion (provider quirk, dropped response_format, or parsed field unused), the model is not even asked in the prompt to emit JSON. That matches the live prose answer.
3. chat() is typed as Optional[str]
def chat(self, prompt: str, ..., output_pydantic: Optional[Any] = None, ...) -> Optional[str]:
The type hint documents the bug. A parameter named output_pydantic whose return type cannot be a Pydantic model is an API lie.
4. Hierarchical manager was already taught the lesson
process/manager_schema.py in 1.7.0 uses ConfigDict(extra="forbid") and parses into ManagerInstructions. User-level output_pydantic did not get the same treatment. The SDK is inconsistent with itself.
Expected vs unexpected (table)
| Step | Expected | Unexpected (live) |
|---|---|---|
Build response_format |
OpenAI-strict JSON schema for Pair |
Happens internally; unused at return |
| Prompt | Schema in response_format or explicit JSON instruction |
Native on → instruction stripped |
| LLM content | {"city":"Paris","country":"France"} |
"The capital of France is Paris." |
| Parse | Pair.model_validate_json(...) |
skipped |
| Return | Pair |
str |
| Failure mode | ValidationError / retry |
silent success |
| Tests | assert type + fields | none covering this live path |
Impact analysis
| Stakeholder | Impact |
|---|---|
| SDK users | Typed pipelines crash one hop later |
| Eval authors | False-green if they only check bool(result) |
| AgentFlow / teams | Cannot pass structured fields between steps without regex |
| OpenAI spend | Extra retries when callers wrap their own JSON repair |
| Trust | README "Structured Output Agents" is not the live chat() path |
Severity justification
| Factor | Assessment |
|---|---|
| Functional correctness | Broken contract — wrong type, wrong shape |
| User visibility | Low — no exception |
| Fix complexity | Moderate — parse + retry already exist for manager / guardrails |
| Scope | Every output_pydantic / output_json chat()/achat() caller |
Recommended architecture
if schema_model:
response_text = ...
try:
data = json.loads(response_text)
parsed = schema_model.model_validate(data)
except (json.JSONDecodeError, ValidationError) as exc:
# retry with explicit JSON instruction (existing guardrail retry pattern)
raise StructuredOutputError(...) from exc
return parsed
Also:
- Use
response.choices[0].message.parsedwhen the OpenAI parse/beta path is used (already done forReflectionOutput). - Keep prompt JSON instruction if native
response_formatis None or LiteLLM dropped it. - Change return type to
Union[str, BaseModel, None]or a smallChatResultwith.textand.parsed. - Fail loud after N retries. Never return prose when
output_pydanticwas requested.
Minimal code fix (chat_mixin.py, both sync and async blocks)
if output_json or output_pydantic:
schema_model = output_pydantic or output_json
parsed = None
if hasattr(schema_model, "model_validate_json") and response_text:
try:
parsed = schema_model.model_validate_json(response_text)
except Exception:
parsed = None
if parsed is None and hasattr(schema_model, "model_validate"):
try:
parsed = schema_model.model_validate(json.loads(response_text))
except Exception:
parsed = None
if parsed is None:
raise ValueError(
f"Agent {self.name}: output_pydantic={schema_model.__name__} "
f"was requested but the model returned unparseable content: "
f"{response_text[:200]!r}"
)
self._append_to_chat_history({"role": "assistant", "content": response_text})
self._persist_message("assistant", response_text)
return parsed
Wire the same into _achat_impl (there is a second copy of the "return raw response" comment on origin/main).
Testing strategy
Live E2E (must hit a real provider — this bug is invisible under mocks that return JSON strings the caller never type-checks):
def test_chat_output_pydantic_returns_model():
class Pair(BaseModel):
city: str
country: str
agent = Agent(instructions="Structured only.", llm="gpt-4o-mini")
out = agent.chat("Capital of France.", output_pydantic=Pair)
assert isinstance(out, Pair)
assert out.city.lower() == "paris"
assert "france" in out.country.lower()
Unit:
- Mock
_chat_completionreturning prose → must raise, not returnstr. - Mock returning valid JSON → must return
Pair. use_native_format=Truewith emptyparsedfield still parsescontent.
Acceptance criteria
-
agent.chat(..., output_pydantic=M)returns an instance ofMon success. - Invalid content raises a dedicated error after retries (not a silent string).
- Sync and async paths behave the same.
- Live test against OpenAI
gpt-4o-miniis in CI or a documented nightly E2E. -
chat()type hints match the contract. - README structured-output example uses this API and is executed in CI.
Workaround (today)
Do not trust output_pydantic on chat(). Parse yourself:
import json
from pydantic import BaseModel
raw = agent.chat("Capital of France. Reply JSON: {\"city\",\"country\"}.")
# still may be prose — add your own retry
Or use Task(output_pydantic=...) if that path still parses (verify separately — do not assume).
Maintainer notes
- Do not close as "works with Task.output_pydantic" without a live
Agent.chattest. The advertised SDK surface ischat/start. - Do not treat this as docs-only. The implementation comment says "return raw response".
- Related (do not conflate): PAI-SDK-009 hierarchical schema — already fixed in 1.7.0; live hierarchical E2E on 2026-08-25 returned
"4"correctly.
SDK improvement value
Fixing this unlocks typed agent pipelines (evals, multi-step handoffs, API gateways) without every consumer re-implementing JSON repair. Aligns Agent.chat() with hierarchical manager parsing already shipped in 1.7.x.
Verification commands
git checkout v4.7.11
pip install -e src/praisonai-agents -e src/praisonai
export OPENAI_API_KEY=...
python -c "
from pydantic import BaseModel, Field
from praisonaiagents import Agent
class Pair(BaseModel):
city: str = Field(...)
country: str = Field(...)
out = Agent(instructions='Structured only.', llm='gpt-4o-mini').chat('Capital of France.', output_pydantic=Pair)
assert isinstance(out, Pair), (type(out), out)
"
- Lingua principale
- Python
- Stelle
- 9.3k
- Fork
- 1.5k
- Merge medio
- 1g 5h
- PR unite (30g)
- 520
Preparare l'ambiente
Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di MervinPraison/PraisonAI
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
MervinPraison/PraisonAI#6710 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
claude
Difficoltà 2/5 1-3 ore Idoneità per principianti 74/100
MervinPraison/PraisonAI#6707 ·
I maintainer di solito rispondono entro 1 giorno
-
bug claude
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 90/100
MervinPraison/PraisonAI#6671 ·
I maintainer di solito rispondono entro 1 giorno
-
Compaction budget regression fixture remains below budget with accurate tokenizationForse di nuovo libera Una pull request per questa issue è stata chiusa senza essere unita. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 62/100
MervinPraison/PraisonAI#5418 · 3 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
bug claude documentation
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
MervinPraison/PraisonAI#5341 · 3 commenti ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di MervinPraison/PraisonAI
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
I maintainer di solito rispondono entro 2 giorni
-
[Bug]: LANGSMITH_SAMPLING_RATE ignores fractional values (isdigit check), traces 100% of requestsAperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 84/100
I maintainer di solito rispondono entro 1 giorno
-
platform/backend tests
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
Significant-Gravitas/AutoGPT#17100 ·
I maintainer di solito rispondono entro 1 giorno
-
bug good first issue
Difficoltà 2/5 1-3 ore Idoneità per principianti 64/100
-
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 61/100
scverse/infercnvpy#238 ·
I maintainer di solito rispondono entro 1 giorno