Problem with resolution

Reproduisez le problème avec les commandes CLI python-inspector et les appels à resolver_api fournis, en utilisant re.txt et l’archive tar setup.py de PyHEADTAIL. Commencez par la stack trace dans python_inspector/api.py et python_inspector/resolution.py autour de get_requirements_from_python_manifest. Le travail est terminé lorsque les requirements invalides n’échouent pas silencieusement, que le comportement de la résolution de setup.py est expliqué ou corrigé, et que la résolution partielle ainsi que la gestion des erreurs sont couvertes par des tests.

There are a couple issues:
1. this file is not a valid requirements file IMHO and we do not return any error:
I save this in re.txt and ran:
python-inspector --json - -p 3.10 -o linux --version -r re.txt

numpy>=1.0
scipy
xobjects
xpart
xdeps

[tests]
cpymad
PyHEADTAIL
pytest

This was copied by mistake as is from the setup.py of xtrack-0.23.2.tar.gz from https://pypi.org/project/xtrack/#files

2. trying to inspect it yields this error which is not right becuase this setup.py is perfectly parsable:

>>> resolver_api(requirement_files=["re.txt"], python_version="310", operating_system="linux")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/python_inspector/api.py", line 254, in resolve_dependencies
    resolution, purls = resolve(
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/python_inspector/api.py", line 313, in resolve
    resolved_dependencies, packages = get_resolved_dependencies(
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/python_inspector/api.py", line 351, in get_resolved_dependencies
    resolver_results = resolver.resolve(requirements=requirements, max_rounds=max_rounds)
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 521, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 402, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 238, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 228, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/python_inspector/resolution.py", line 600, in get_dependencies
    return list(self._iter_dependencies(candidate))
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/python_inspector/resolution.py", line 591, in _iter_dependencies
    for r in self.get_requirements_for_package(purl=purl, candidate=candidate):
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/python_inspector/resolution.py", line 470, in get_requirements_for_package_from_pypi_simple
    yield from get_requirements_from_python_manifest(
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/python_inspector/resolution.py", line 304, in get_requirements_from_python_manifest
    raise Exception(
Exception: Unable to collect setup.py dependencies securely: .cache/thirdparty/extracted_sdists/PyHEADTAIL-1.16.0/PyHEADTAIL-1.16.0/setup.py

3. Trying to call the API with the setup.py is not right:
See https://files.pythonhosted.org/packages/7c/dd/80d8ec272179941bf6cc181446127da194eb82b7138549a6939583c4c318/PyHEADTAIL-1.16.0.tar.gz (which should have been in the graph above) returns an imcomplete deps tree (or rather nothing)

>>> resolver_api(setup_py_file=".cache/thirdparty/extracted_sdists/PyHEADTAIL-1.16.0/PyHEADTAIL-1.16.0/setup.py", python_version="310", operating_system="linux")
Resolution(resolution=[], packages=[], files=[{'type': 'file', 'path': '.cache/thirdparty/extracted_sdists/PyHEADTAIL-1.16.0/PyHEADTAIL-1.16.0/setup.py', 'package_data': [{'type': 'pypi', 'namespace': None, 'name': 'PyHEADTAIL', 'version': '1.16.0', 'qualifiers': {}, 'subpath': None, 'primary_language': 'Python', 'description': 'CERN PyHEADTAIL numerical n-body simulation code for simulating macro-particle beam dynamics with collective effects.', 'release_date': None, 'parties': [{'type': 'person', 'role': 'author', 'name': 'Kevin Li', 'email': '[email protected]', 'url': None}, {'type': 'person', 'role': 'maintainer', 'name': 'Adrian Oeftiger', 'email': '[email protected]', 'url': None}], 'keywords': [], 'homepage_url': 'https://github.com/PyCOMPLETE/PyHEADTAIL', 'download_url': None, 'size': None, 'sha1': None, 'md5': None, 'sha256': None, 'sha512': None, 'bug_tracking_url': None, 'code_view_url': None, 'vcs_url': None, 'copyright': None, 'license_expression': None, 'declared_license': {}, 'notice_text': None, 'source_packages': [], 'file_references': [], 'extra_data': {}, 'dependencies': [{'purl': 'pkg:pypi/h5py', 'extracted_requirement': 'h5py', 'scope': 'setup', 'is_runtime': True, 'is_optional': False, 'is_resolved': False, 'resolved_package': {}, 'extra_data': {}}, {'purl': 'pkg:pypi/numpy', 'extracted_requirement': 'numpy', 'scope': 'setup', 'is_runtime': True, 'is_optional': False, 'is_resolved': False, 'resolved_package': {}, 'extra_data': {}}, {'purl': 'pkg:pypi/scipy', 'extracted_requirement': 'scipy', 'scope': 'setup', 'is_runtime': True, 'is_optional': False, 'is_resolved': False, 'resolved_package': {}, 'extra_data': {}}, {'purl': 'pkg:pypi/cython', 'extracted_requirement': 'cython', 'scope': 'setup', 'is_runtime': True, 'is_optional': False, 'is_resolved': False, 'resolved_package': {}, 'extra_data': {}}], 'repository_homepage_url': 'https://pypi.org/project/PyHEADTAIL', 'repository_download_url': 'https://pypi.org/packages/source/P/PyHEADTAIL/PyHEADTAIL-1.16.0.tar.gz', 'api_data_url': 'https://pypi.org/pypi/PyHEADTAIL/1.16.0/json', 'datasource_id': 'pypi_setup_py', 'purl': 'pkg:pypi/[email protected]'}]}])

4. Scanning that seame setup.py with SCTK works without a glitch with these deps

                dependencies:
                    -   purl: pkg:pypi/h5py
                        extracted_requirement: h5py
                        scope: setup
                        is_runtime: yes
                        is_optional: no
                        is_resolved: no
                        resolved_package: {}
                        extra_data: {}
                    -   purl: pkg:pypi/numpy
                        extracted_requirement: numpy
                        scope: setup
                        is_runtime: yes
                        is_optional: no
                        is_resolved: no
                        resolved_package: {}
                        extra_data: {}
                    -   purl: pkg:pypi/scipy
                        extracted_requirement: scipy
                        scope: setup
                        is_runtime: yes
                        is_optional: no
                        is_resolved: no
                        resolved_package: {}
                        extra_data: {}
                    -   purl: pkg:pypi/cython
                        extracted_requirement: cython
                        scope: setup
                        is_runtime: yes
                        is_optional: no
                        is_resolved: no
                        resolved_package: {}
                        extra_data: {}

5. running this does not have the extra requires:
python-inspector --spec "xtrack==0.23.3" --json - -p 3.10 -o linux

"resolved_dependencies_graph": [
    {
      "package": "pkg:pypi/[email protected]",
      "dependencies": [
        "pkg:pypi/[email protected]"
      ]
    },
    {
      "package": "pkg:pypi/[email protected]",
      "dependencies": []
    },
    {
      "package": "pkg:pypi/[email protected]",
      "dependencies": []
    },
    {
      "package": "pkg:pypi/[email protected]",
      "dependencies": []
    },
    {
      "package": "pkg:pypi/[email protected]",
      "dependencies": [
        "pkg:pypi/[email protected]"
      ]
    },
    {
      "package": "pkg:pypi/[email protected]",
      "dependencies": [
        "pkg:pypi/[email protected]"
      ]
    },
    {
      "package": "pkg:pypi/[email protected]",
      "dependencies": [
        "pkg:pypi/[email protected]",
        "pkg:pypi/[email protected]"
      ]
    },
    {
      "package": "pkg:pypi/[email protected]",
      "dependencies": [
        "pkg:pypi/[email protected]",
        "pkg:pypi/[email protected]",
        "pkg:pypi/[email protected]"
      ]
    },
    {
      "package": "pkg:pypi/[email protected]",
      "dependencies": [
        "pkg:pypi/[email protected]",
        "pkg:pypi/[email protected]",
        "pkg:pypi/[email protected]",
        "pkg:pypi/[email protected]",
        "pkg:pypi/[email protected]"
      ]
    }
  ]

6. we should have a way to return partial resolutions and we should not fail silently