`EvaluationResult.feedback_config` silently drops unknown or partial dict fields without error · langchain-ai/langchain#31802

(16 comments) (1 reaction) (0 assignees)Python (136,758 stars) (22,617 forks)batch import

bugcoredocumentationexternalhelp wanted

Description

Checked other resources

I added a very descriptive title to this issue.
I used the GitHub search to find a similar question and didn't find it.
I am sure that this is a bug in LangChain rather than my code.
The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).
I posted a self-contained, minimal, reproducible example. A maintainer can copy it and run it AS IS.

Example Code

from langchain_core.tracers.evaluation import EvaluationResult

result = EvaluationResult(
    key="sentiment",
    value="positive",
    feedback_config={"threshold": 1.0}
)
print(result.feedback_config)

Error Message and Stack Trace (if applicable)

{}

Description

When passing a dict to EvaluationResult.feedback_config, the data is silently discarded. This causes unexpected behavior and makes it impossible to store arbitrary dict configurations directly, despite the type hint allowing dict.

System Info

System Information

OS: Linux OS Version: #148-Ubuntu SMP Fri Mar 14 19:05:48 UTC 2025 Python Version: 3.10.16 (main, Dec 11 2024, 16:24:50) [GCC 11.2.0]

Package Information

langchain_core: 0.3.65 langchain: 0.3.25 langchain_community: 0.3.25 langsmith: 0.3.45 langchain_anthropic: 0.3.1 langchain_aws: 0.2.2 langchain_chroma: 0.2.0 langchain_cohere: 0.3.5 langchain_experimental: 0.3.2 langchain_fireworks: 0.2.6 langchain_google_vertexai: 2.0.5 langchain_groq: 0.2.2 langchain_mistralai: 0.2.4 langchain_openai: 0.2.14 langchain_text_splitters: 0.3.8 langchain_together: 0.2.0 langchain_unstructured: 0.1.5 langgraph_sdk: 0.1.34

Optional packages not installed

langserve

Other Dependencies

aiohttp: 3.12.12 aiohttp<4.0.0,>=3.8.3: Installed. No version info available. anthropic: 0.40.0 anthropic[vertexai]: Installed. No version info available. async-timeout<5.0.0,>=4.0.0;: Installed. No version info available. boto3: 1.35.42 chromadb: 0.5.23 cohere: 5.15.0 dataclasses-json<0.7,>=0.5.7: Installed. No version info available. defusedxml: 0.7.1 fastapi: 0.115.9 fireworks-ai: 0.15.7 google-cloud-aiplatform: 1.70.0 google-cloud-storage: 2.18.2 groq: 0.11.0 httpx: 0.28.1 httpx-sse: 0.4.0 httpx-sse<1.0.0,>=0.4.0: Installed. No version info available. jsonpatch<2.0,>=1.33: Installed. No version info available. langchain-anthropic;: Installed. No version info available. langchain-aws;: Installed. No version info available. langchain-azure-ai;: Installed. No version info available. langchain-cohere;: Installed. No version info available. langchain-community;: Installed. No version info available. langchain-core<1.0.0,>=0.3.51: Installed. No version info available. langchain-core<1.0.0,>=0.3.58: Installed. No version info available. langchain-core<1.0.0,>=0.3.65: Installed. No version info available. langchain-deepseek;: Installed. No version info available. langchain-fireworks;: Installed. No version info available. langchain-google-genai;: Installed. No version info available. langchain-google-vertexai;: Installed. No version info available. langchain-groq;: Installed. No version info available. langchain-huggingface;: Installed. No version info available. langchain-mistralai;: Installed. No version info available. langchain-ollama;: Installed. No version info available. langchain-openai;: Installed. No version info available. langchain-perplexity;: Installed. No version info available. langchain-text-splitters<1.0.0,>=0.3.8: Installed. No version info available. langchain-together;: Installed. No version info available. langchain-xai;: Installed. No version info available. langchain<1.0.0,>=0.3.25: Installed. No version info available. langsmith-pyo3: Installed. No version info available. langsmith<0.4,>=0.1.125: Installed. No version info available. langsmith<0.4,>=0.1.17: Installed. No version info available. langsmith<0.4,>=0.3.45: Installed. No version info available. numpy: 1.26.4 numpy>=1.26.2;: Installed. No version info available. numpy>=2.1.0;: Installed. No version info available. openai: 1.86.0 openai-agents: Installed. No version info available. opentelemetry-api: 1.34.1 opentelemetry-exporter-otlp-proto-http: 1.34.1 opentelemetry-sdk: 1.34.1 orjson: 3.10.16 packaging: 24.1 packaging<25,>=23.2: Installed. No version info available. pandas: 2.2.3 pydantic: 2.11.1 pydantic-settings<3.0.0,>=2.4.0: Installed. No version info available. pydantic<3.0.0,>=2.7.4: Installed. No version info available. pydantic>=2.7.4: Installed. No version info available. pytest: 8.3.4 PyYAML>=5.3: Installed. No version info available. requests: 2.32.3 requests-toolbelt: 1.0.0 requests<3,>=2: Installed. No version info available. rich: 13.9.3 SQLAlchemy<3,>=1.4: Installed. No version info available. tabulate: 0.9.0 tenacity!=8.4.0,<10,>=8.1.0: Installed. No version info available. tenacity!=8.4.0,<10.0.0,>=8.1.0: Installed. No version info available. tiktoken: 0.8.0 tokenizers: 0.21.1 typing-extensions>=4.7: Installed. No version info available. unstructured-client: 0.25.9 unstructured[all-docs]: Installed. No version info available. zstandard: 0.23.0

Contributor guide

Tech stack: python
Domain: backendapi
Issue type: bug
Difficulty: 2
Estimated time: 1-3 hours
Activity status: active
Clarity: clear
Prerequisites: Pythonpydantic basics
Newbie friendliness: 80
Research direction: Investigate the EvaluationResult class in langchain core/tracers/evaluation.py to understand how feedback config is handled. The current implementation likely uses pydantic validation that ignores extra fields. The fix should either raise a ValueError for unknown keys or store the full dict. Check if there is a model config with extra='forbid' or similar. Look at the pydantic model definition and tests in tests/unit tests/tracers/test evaluation.py.