openapi_core.create_spec fails to resolve $ref to parent directory

Reproduce the failure with create_spec in the provided example and the three relative-reference YAML files. Start with openapi_core/schema/shortcuts.py and the dereferencing path in openapi_core/schema/schemas/factories.py, then inspect openapi_spec_validator/validators.py and handlers.py. Done means references from sub/paths.yaml resolve schemas.yaml without incorrect path prefixes.

I have a schema with a reference graph that looks like openapi.yaml -> sub/paths.yaml -> schemas.yaml. The middle schema is in a subdirectory and the first and last schemas are in the same directory. This fails to resolve correctly when proving the spec_dict generated from openapi.yaml to create_spec in the following way:

from pathlib import Path

from openapi_core import create_spec
from openapi_spec_validator.schemas import read_yaml_file

path = Path("relative-reference/openapi.yaml").resolve()
create_spec(read_yaml_file(path), spec_url=path.as_uri())

The three OpenAPI files are as follows:

relative-reference/openapi.yaml:

openapi: "3.0.0"

info:
  version: "1.0.0"
  title: Relative references not resolving correctly

paths:
  /:
    $ref: "./sub/paths.yaml#/root"

relative-reference/sub/paths.yaml:

root:
  get:
    summary: Some API endpoint
    responses:
      '200':
        description: Content for the API
        content:
          application/json:
            schema:
              type: array
              items:
                $ref: ../schemas.yaml#/Item

relative-reference/schemas.yaml:

Item:
  type: object
  required: ["name"]
  properties:
    name:
      type: string
      maxLength: 40

The error I get ends with

  File "/workdir/.pip/lib/python3.7/site-packages/jsonschema/validators.py", line 754, in resolving
    url, resolved = self.resolve(ref)
  File "/workdir/.pip/lib/python3.7/site-packages/jsonschema/validators.py", line 766, in resolve
    return url, self._remote_cache(url)
  File "/workdir/.pip/lib/python3.7/site-packages/jsonschema/validators.py", line 779, in resolve_from_url
    raise exceptions.RefResolutionError(exc)
jsonschema.exceptions.RefResolutionError: <urlopen error [Errno 2] No such file or directory: '/workdir/schemas.yaml'>

The path it tries to resolve lacks the relative-reference directory name.

However, fixing it by adding the missing path portion in the reference does not help. When changing the parent-directory reference to ../relative-reference/schemas.yaml#/Item, the error changes but remains:

jsonschema.exceptions.RefResolutionError: <urlopen error [Errno 2] No such file or directory: '/workdir/relative-reference/relative-reference/schemas.yaml'>

The 'base' directory was left on the path and now we have a failure because we try to access a directory that doesn't exist. Adding a differently named path portion here results in the same problem (except the failed path will be e.g. /workdir/relative-reference/dummy/schemas.yaml. This error happens on all Pythons 3.6 through 3.9.

The full track trace:

Traceback (most recent call last):
  File "/workdir/.pip/lib/python3.7/site-packages/jsonschema/validators.py", line 774, in resolve_from_url
    document = self.store[url]
  File "/workdir/.pip/lib/python3.7/site-packages/jsonschema/_utils.py", line 22, in __getitem__
    return self.store[self.normalize(uri)]
KeyError: 'file:///workdir/schemas.yaml'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/urllib/request.py", line 1506, in open_local_file
    stats = os.stat(localfile)
FileNotFoundError: [Errno 2] No such file or directory: '/workdir/schemas.yaml'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workdir/.pip/lib/python3.7/site-packages/jsonschema/validators.py", line 777, in resolve_from_url
    document = self.resolve_remote(url)
  File "/workdir/.pip/lib/python3.7/site-packages/jsonschema/validators.py", line 856, in resolve_remote
    result = self.handlers[scheme](uri)
  File "/workdir/.pip/lib/python3.7/site-packages/openapi_spec_validator/handlers.py", line 35, in __call__
    f = urlopen(url, timeout=timeout)
  File "/usr/local/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python3.7/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/local/lib/python3.7/urllib/request.py", line 543, in _open
    '_open', req)
  File "/usr/local/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.7/urllib/request.py", line 1484, in file_open
    return self.open_local_file(req)
  File "/usr/local/lib/python3.7/urllib/request.py", line 1523, in open_local_file
    raise URLError(exp)
urllib.error.URLError: <urlopen error [Errno 2] No such file or directory: '/workdir/schemas.yaml'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "example.py", line 7, in <module>
    create_spec(read_yaml_file(path), spec_url=path.as_uri())
  File "/workdir/.pip/lib/python3.7/site-packages/openapi_core/schema/shortcuts.py", line 20, in create_spec
    return spec_factory.create(spec_dict, spec_url=spec_url)
  File "/workdir/.pip/lib/python3.7/site-packages/openapi_core/schema/specs/factories.py", line 49, in create
    info, list(paths), servers=list(servers), components=components,
  File "/workdir/.pip/lib/python3.7/site-packages/openapi_core/schema/paths/generators.py", line 38, in generate
    path_name, list(operations), parameters=list(parameters),
  File "/workdir/.pip/lib/python3.7/site-packages/openapi_core/schema/operations/generators.py", line 77, in generate
    servers=list(servers), extensions=extensions,
  File "/workdir/.pip/lib/python3.7/site-packages/openapi_core/schema/operations/models.py", line 16, in __init__
    self.responses = dict(responses)
  File "/workdir/.pip/lib/python3.7/site-packages/openapi_core/schema/responses/generators.py", line 40, in generate
    extensions=extensions,
  File "/workdir/.pip/lib/python3.7/site-packages/openapi_core/schema/responses/models.py", line 15, in __init__
    self.content = content and Content(content) or Content()
  File "/workdir/.pip/lib/python3.7/site-packages/openapi_core/schema/media_types/generators.py", line 30, in generate
    schema, _ = self.schemas_registry.get_or_create(schema_spec)
  File "/workdir/.pip/lib/python3.7/site-packages/openapi_core/schema/schemas/registries.py", line 28, in get_or_create
    schema = self.create(schema_deref)
  File "/workdir/.pip/lib/python3.7/site-packages/openapi_core/schema/schemas/factories.py", line 68, in create
    items = self._create_items(items_spec)
  File "/workdir/.pip/lib/python3.7/site-packages/openapi_core/schema/schemas/factories.py", line 101, in _create_items
    return self.create(items_spec)
  File "/workdir/.pip/lib/python3.7/site-packages/openapi_core/schema/schemas/factories.py", line 21, in create
    schema_deref = self.dereferencer.dereference(schema_spec)
  File "/workdir/.pip/lib/python3.7/site-packages/openapi_spec_validator/validators.py", line 36, in dereference
    with resolver.resolving(ref) as target:
  File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/workdir/.pip/lib/python3.7/site-packages/jsonschema/validators.py", line 754, in resolving
    url, resolved = self.resolve(ref)
  File "/workdir/.pip/lib/python3.7/site-packages/jsonschema/validators.py", line 766, in resolve
    return url, self._remote_cache(url)
  File "/workdir/.pip/lib/python3.7/site-packages/jsonschema/validators.py", line 779, in resolve_from_url
    raise exceptions.RefResolutionError(exc)
jsonschema.exceptions.RefResolutionError: <urlopen error [Errno 2] No such file or directory: '/workdir/schemas.yaml'>

Relevant package versions:

jsonschema==3.2.0
openapi-core==0.13.7
openapi-schema-validator==0.1.2
openapi-spec-validator==0.2.9