Problem with resolution

Tái hiện vấn đề bằng các lệnh CLI python-inspector và các lệnh gọi resolver_api được cung cấp, sử dụng re.txt và tarball setup.py của PyHEADTAIL. Bắt đầu từ stack trace trong python_inspector/api.py và python_inspector/resolution.py quanh get_requirements_from_python_manifest. Hoàn thành khi các requirements không hợp lệ không thất bại một cách im lặng, hành vi phân giải setup.py được giải thích hoặc sửa, và việc phân giải từng phần cũng như xử lý lỗi được bao phủ bởi các bài kiểm thử.

There are a couple issues:
1. this file is not a valid requirements file IMHO and we do not return any error:
I save this in re.txt and ran:
python-inspector --json - -p 3.10 -o linux --version -r re.txt

numpy>=1.0
scipy
xobjects
xpart
xdeps

[tests]
cpymad
PyHEADTAIL
pytest

This was copied by mistake as is from the setup.py of xtrack-0.23.2.tar.gz from https://pypi.org/project/xtrack/#files

2. trying to inspect it yields this error which is not right becuase this setup.py is perfectly parsable:

>>> resolver_api(requirement_files=["re.txt"], python_version="310", operating_system="linux")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/python_inspector/api.py", line 254, in resolve_dependencies
    resolution, purls = resolve(
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/python_inspector/api.py", line 313, in resolve
    resolved_dependencies, packages = get_resolved_dependencies(
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/python_inspector/api.py", line 351, in get_resolved_dependencies
    resolver_results = resolver.resolve(requirements=requirements, max_rounds=max_rounds)
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 521, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 402, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 238, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 228, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/python_inspector/resolution.py", line 600, in get_dependencies
    return list(self._iter_dependencies(candidate))
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/python_inspector/resolution.py", line 591, in _iter_dependencies
    for r in self.get_requirements_for_package(purl=purl, candidate=candidate):
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/python_inspector/resolution.py", line 470, in get_requirements_for_package_from_pypi_simple
    yield from get_requirements_from_python_manifest(
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/python_inspector/resolution.py", line 304, in get_requirements_from_python_manifest
    raise Exception(
Exception: Unable to collect setup.py dependencies securely: .cache/thirdparty/extracted_sdists/PyHEADTAIL-1.16.0/PyHEADTAIL-1.16.0/setup.py

3. Trying to call the API with the setup.py is not right:
See https://files.pythonhosted.org/packages/7c/dd/80d8ec272179941bf6cc181446127da194eb82b7138549a6939583c4c318/PyHEADTAIL-1.16.0.tar.gz (which should have been in the graph above) returns an imcomplete deps tree (or rather nothing)

>>> resolver_api(setup_py_file=".cache/thirdparty/extracted_sdists/PyHEADTAIL-1.16.0/PyHEADTAIL-1.16.0/setup.py", python_version="310", operating_system="linux")
Resolution(resolution=[], packages=[], files=[{'type': 'file', 'path': '.cache/thirdparty/extracted_sdists/PyHEADTAIL-1.16.0/PyHEADTAIL-1.16.0/setup.py', 'package_data': [{'type': 'pypi', 'namespace': None, 'name': 'PyHEADTAIL', 'version': '1.16.0', 'qualifiers': {}, 'subpath': None, 'primary_language': 'Python', 'description': 'CERN PyHEADTAIL numerical n-body simulation code for simulating macro-particle beam dynamics with collective effects.', 'release_date': None, 'parties': [{'type': 'person', 'role': 'author', 'name': 'Kevin Li', 'email': '[email protected]', 'url': None}, {'type': 'person', 'role': 'maintainer', 'name': 'Adrian Oeftiger', 'email': '[email protected]', 'url': None}], 'keywords': [], 'homepage_url': 'https://github.com/PyCOMPLETE/PyHEADTAIL', 'download_url': None, 'size': None, 'sha1': None, 'md5': None, 'sha256': None, 'sha512': None, 'bug_tracking_url': None, 'code_view_url': None, 'vcs_url': None, 'copyright': None, 'license_expression': None, 'declared_license': {}, 'notice_text': None, 'source_packages': [], 'file_references': [], 'extra_data': {}, 'dependencies': [{'purl': 'pkg:pypi/h5py', 'extracted_requirement': 'h5py', 'scope': 'setup', 'is_runtime': True, 'is_optional': False, 'is_resolved': False, 'resolved_package': {}, 'extra_data': {}}, {'purl': 'pkg:pypi/numpy', 'extracted_requirement': 'numpy', 'scope': 'setup', 'is_runtime': True, 'is_optional': False, 'is_resolved': False, 'resolved_package': {}, 'extra_data': {}}, {'purl': 'pkg:pypi/scipy', 'extracted_requirement': 'scipy', 'scope': 'setup', 'is_runtime': True, 'is_optional': False, 'is_resolved': False, 'resolved_package': {}, 'extra_data': {}}, {'purl': 'pkg:pypi/cython', 'extracted_requirement': 'cython', 'scope': 'setup', 'is_runtime': True, 'is_optional': False, 'is_resolved': False, 'resolved_package': {}, 'extra_data': {}}], 'repository_homepage_url': 'https://pypi.org/project/PyHEADTAIL', 'repository_download_url': 'https://pypi.org/packages/source/P/PyHEADTAIL/PyHEADTAIL-1.16.0.tar.gz', 'api_data_url': 'https://pypi.org/pypi/PyHEADTAIL/1.16.0/json', 'datasource_id': 'pypi_setup_py', 'purl': 'pkg:pypi/[email protected]'}]}])

4. Scanning that seame setup.py with SCTK works without a glitch with these deps

                dependencies:
                    -   purl: pkg:pypi/h5py
                        extracted_requirement: h5py
                        scope: setup
                        is_runtime: yes
                        is_optional: no
                        is_resolved: no
                        resolved_package: {}
                        extra_data: {}
                    -   purl: pkg:pypi/numpy
                        extracted_requirement: numpy
                        scope: setup
                        is_runtime: yes
                        is_optional: no
                        is_resolved: no
                        resolved_package: {}
                        extra_data: {}
                    -   purl: pkg:pypi/scipy
                        extracted_requirement: scipy
                        scope: setup
                        is_runtime: yes
                        is_optional: no
                        is_resolved: no
                        resolved_package: {}
                        extra_data: {}
                    -   purl: pkg:pypi/cython
                        extracted_requirement: cython
                        scope: setup
                        is_runtime: yes
                        is_optional: no
                        is_resolved: no
                        resolved_package: {}
                        extra_data: {}

5. running this does not have the extra requires:
python-inspector --spec "xtrack==0.23.3" --json - -p 3.10 -o linux

"resolved_dependencies_graph": [
    {
      "package": "pkg:pypi/[email protected]",
      "dependencies": [
        "pkg:pypi/[email protected]"
      ]
    },
    {
      "package": "pkg:pypi/[email protected]",
      "dependencies": []
    },
    {
      "package": "pkg:pypi/[email protected]",
      "dependencies": []
    },
    {
      "package": "pkg:pypi/[email protected]",
      "dependencies": []
    },
    {
      "package": "pkg:pypi/[email protected]",
      "dependencies": [
        "pkg:pypi/[email protected]"
      ]
    },
    {
      "package": "pkg:pypi/[email protected]",
      "dependencies": [
        "pkg:pypi/[email protected]"
      ]
    },
    {
      "package": "pkg:pypi/[email protected]",
      "dependencies": [
        "pkg:pypi/[email protected]",
        "pkg:pypi/[email protected]"
      ]
    },
    {
      "package": "pkg:pypi/[email protected]",
      "dependencies": [
        "pkg:pypi/[email protected]",
        "pkg:pypi/[email protected]",
        "pkg:pypi/[email protected]"
      ]
    },
    {
      "package": "pkg:pypi/[email protected]",
      "dependencies": [
        "pkg:pypi/[email protected]",
        "pkg:pypi/[email protected]",
        "pkg:pypi/[email protected]",
        "pkg:pypi/[email protected]",
        "pkg:pypi/[email protected]"
      ]
    }
  ]

6. we should have a way to return partial resolutions and we should not fail silently