fix(adapter): honor custom response backends during preparation and serialization

Start with the response preparation in src/azure_functions_validation/decorator.py and response dispatch in pipeline.py, then compare the public contract in adapter.py and docs/adapters.md. Review the existing conformance tests and the response serialization protections referenced in #463 before choosing an approach. Done means decorated-handler tests cover custom non-Pydantic backends and sync/async handlers while preserving declared-model serialization and error handling.

Context

Reviewed commit: b38d702adb1039ed1eda2ef3799d873af2506f41.
Reproduced with Python 3.12.14, Pydantic 2.14.1, and azure-functions 1.25.0.

The public adapter guide says a custom ValidationAdapter can serve a non-Pydantic model system, and defines serialize(obj) as accepting the validated handler result and returning its body/content type. The response-model pipeline does not honor that contract:

1. Decoration unconditionally constructs a Pydantic TypeAdapter(response_model), so a custom backend's ordinary model class raises before that adapter can validate it.
2. When the declared model happens to be Pydantic-compatible, the custom adapter validates the result but its serializer is never called. The pipeline unconditionally uses Pydantic JSON serialization and application/json instead.

The conformance suite introduced through #509 tests adapter methods in isolation, so an adapter can satisfy that surface without its response backend being usable through @validate_http.

Reproduction A: non-Pydantic model cannot reach its adapter

This example reuses the built-in request methods and supplies an independent response backend; no request model is configured.

from azure_functions_validation import PydanticAdapter, validate_http

class NativeRecord:
    def __init__(self, value):
        self.value = value

class NativeAdapter(PydanticAdapter):
    def __init__(self):
        super().__init__()
        self.validation_calls = 0

    def validate_response(self, obj, model, *, type_adapter=None):
        self.validation_calls += 1
        return obj

    def serialize(self, obj):
        return "custom:" + obj.value, "text/plain"

adapter = NativeAdapter()
try:
    @validate_http(adapter=adapter, response_model=NativeRecord)
    def native(req):
        return NativeRecord("ok")
except Exception as exc:
    print(type(exc).__name__)
    print(str(exc).split("\n")[0])
print("adapter validation calls:", adapter.validation_calls)

Observed: decoration raises PydanticSchemaGenerationError: Unable to generate pydantic-core schema for <class ...NativeRecord>; adapter validation calls are zero. In a Function App module, the same import-time failure can prevent the affected app from indexing.

Expected: a supported custom backend can prepare and validate its own model without requiring that model to implement Pydantic's schema protocol. If some response capability is intentionally unsupported, the public adapter contract and capability negotiation must identify that boundary explicitly rather than promising independent backends that cannot be used.

Reproduction B: custom serialization is ignored for a compatible model

from azure.functions import HttpRequest
from pydantic import BaseModel
from azure_functions_validation import PydanticAdapter, validate_http

class Output(BaseModel):
    value: str

class CustomAdapter(PydanticAdapter):
    def __init__(self):
        super().__init__()
        self.validation_calls = 0
        self.serialization_calls = 0

    def validate_response(self, obj, model, *, type_adapter=None):
        self.validation_calls += 1
        return super().validate_response(obj, model, type_adapter=type_adapter)

    def serialize(self, obj):
        self.serialization_calls += 1
        return "custom:" + obj.value, "text/plain"

adapter = CustomAdapter()

@validate_http(adapter=adapter, response_model=Output)
def handler(req):
    return Output(value="ok")

response = handler(HttpRequest(
    method="GET", url="http://localhost/api/x", body=b""
))
print(response.status_code, response.get_body(), dict(response.headers))
print(adapter.validation_calls, adapter.serialization_calls)

Observed: response 200, body b'{"value":"ok"}', Content-Type: application/json, calls 1 0.

Expected under the published adapter contract: the configured backend's supported response serialization path is used and its content/content type are preserved; the custom serializer must not be silently bypassed.

Fix direction: make response preparation and declared-schema serialization backend-aware, and establish one consistent public capability contract between decorator, pipeline, protocol, and conformance tests. Preserve the default Pydantic path's declared-model serialization, including exclusion of subclass-only fields and alias/exclusion options. Simply routing the default Pydantic result through unrestricted runtime-object serialization would regress the data-exposure fix in #463 and is not an acceptable repair.

Acceptance Checklist

• A custom backend with a non-Pydantic response model can be used through the supported decorated-handler path without unconditional Pydantic schema construction.
• Custom response validation and supported serialization are called in order, with the validated result reaching the serializer and its body/content type preserved.
• Backend preparation/reuse and serialization options have an explicit, documented contract; incompatible capabilities fail with actionable configuration guidance.
• Add decorated-handler integration/conformance tests for a non-Pydantic response backend, not only direct calls to adapter methods.
• Cover both synchronous and asynchronous handlers and preserve structured response validation/serialization error handling.
• The default Pydantic backend still serializes against the declared response type, excludes subclass-only fields, and honors aliases and response exclusion options.
• Keep the protections from #463 and the default response serialization regression tests intact.
• Update ValidationAdapter, docs/adapters.md, and reusable conformance expectations together; follow the repository's compatibility/versioning policy if the public protocol changes.
• Repository quality gates and required coverage remain satisfied without adding a second backend implementation as a runtime dependency.

Out of scope

• Shipping or maintaining a new validation backend.
• Changing default Pydantic wire-format behavior or allowing subclass-only fields to leak.
• General adapter exception-boundary repairs, which can be tracked independently.
• Requiring implementation work in sibling repositories.

References

• Reviewed commit
• Unconditional Pydantic response preparation
• Response-model validation and serializer dispatch
• Public adapter methods
• Published custom-backend and serialization contract
• Method-level conformance suite
• Public conformance work
• Declared-response serialization protection to preserve