ignored_dependency_scopes does not exclude npm devDependencies
Los mantenedores suelen responder en 1 día
Evaluación
- Dificultad
- 2/5
- Tiempo estimado
- 1-3 horas
- Aptitud para principiantes
- 74/100
Línea de trabajo
Inspecciona scanpipe/pipes/__init__.py en ignore_dependency_scope() y confirma cómo se extraen los metadatos de las dependencias para el tipo de paquete y el scope. Revisa las pruebas que cubren este flujo del trabajo existente sobre ignored_dependency_scopes para ver las suposiciones actuales y, después, añade o ajusta un caso de regresión para las dependencias de npm que solo proporcionan purl y scope. Ejecuta las pruebas del filtro de dependencias y marca la tarea como completada cuando una regla de ignored_dependency_scopes para package_type: npm y scope: devDependencies ya no cree ni resuelva esas devDependencies.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Description
The ignored_dependency_scopes setting in scancode-config.yml does not appear to exclude npm devDependencies during a ScanCode.io scan.
I encountered this while scanning ScanCode Workbench 4.0.2. I intentionally used the same configuration that is documented in the official ScanCode.io end-to-end tutorial:
product_name: scancode-workbench
product_version: '4.0.2'
ignored_patterns:
- 'tests/*'
ignored_dependency_scopes:
- package_type: npm
scope: devDependencies
The scancode-config.yml is located at the root of the project inside a single uploaded source archive.
The scan completes successfully, but dependencies with the scope devDependencies are still included instead of being ignored.
This seems to affect the ignored_dependency_scopes filtering itself rather than npm dependency detection, since ScanCode Toolkit correctly identifies these dependencies with the scope devDependencies.
How To Reproduce
Steps:
- Use ScanCode Workbench 4.0.2 as the npm project.
- Add the following
scancode-config.ymlto the project root:
product_name: scancode-workbench
product_version: '4.0.2'
ignored_patterns:
- 'tests/*'
ignored_dependency_scopes:
- package_type: npm
scope: devDependencies
- Create a source archive containing the project and configuration.
- Create a ScanCode.io project and upload the archive as a single input.
- Run package inspection/scanning so that npm dependencies from
package-lock.jsonare collected. - Inspect the resulting dependencies.
Dependencies with:
scope: devDependencies
are still present even though they should be ignored by the project configuration.
The provided package-lock.json uses lockfile version 3 and contains a large number of dependencies marked as development dependencies, so the behavior is easy to reproduce.
Expected behavior
npm dependencies with the scope devDependencies should be ignored when the following configuration is present:
ignored_dependency_scopes:
- package_type: npm
scope: devDependencies
According to the ScanCode.io documentation, dependencies matching an ignored package type and scope should not be created or resolved.
This configuration is also the exact example currently used in the ScanCode.io tutorial with ScanCode Workbench 4.0.2.
Actual behavior
npm dependencies with scope: devDependencies are still created/included in the scan results.
Suspected root cause
I looked into the implementation and suspect that the issue is in ignore_dependency_scope() in:
scanpipe/pipes/__init__.py
The function currently retrieves the package type and scope from the dependency data:
dependency_package_type = dependency_data.get("package_type")
dependency_scope = dependency_data.get("scope")
and only evaluates the ignore rule when both values are present.
However, ScanCode Toolkit's DependentPackage / Dependency model does not appear to contain a package_type field.
The dependency model contains fields such as:
purl
extracted_requirement
scope
is_runtime
is_optional
is_pinned
is_direct
resolved_package
extra_data
For an npm dependency, the package type is already represented by the PURL, for example:
pkg:npm/...
while the dependency scope is correctly represented as:
devDependencies
Therefore, for a normal dependency generated by ScanCode Toolkit, this lookup appears to result in:
dependency_data.get("package_type") # None
As a consequence, the following condition is not entered:
if dependency_package_type and dependency_scope:
and the configured ignore rule is never evaluated.
Relation to the original implementation
ignored_dependency_scopes was introduced as part of #1197 and implemented in #1235.
There is another interesting detail in the tests added with #1235: for the positive ignore_dependency_scope() test case, package_type appears to be manually added to the dependency dictionary before calling the function.
In other words, the test verifies the behavior with something similar to:
dependency_data["package_type"] = "pypi"
dependency_data["scope"] = "tests"
This may explain why the implementation passes its unit test even though a normal dependency produced by ScanCode Toolkit does not contain that package_type field.
A regression test using an actual ScanCode Toolkit Dependency / DependentPackage object may expose the problem.
Possible approach
One possible solution could be to derive the package type from the dependency PURL when package_type is not explicitly available.
For example, a dependency such as:
pkg:npm/[email protected]
already provides the required package type (npm).
Alternatively, the dependency data could be normalized so that the package type is populated before ignore_dependency_scope() is called.
The exact implementation is of course up to the maintainers.
Related issues / changes
- #1197 – Enhance and create detailed documentation for
scancode-config.yml - #1235 – Add support for
ignored_dependency_scopesfield for configuration - #1236 – Support storing
scancode-config.ymlfiles in the scanned codebase
#1236 specifically documents that, for a single archive input, a scancode-config.yml located at the root should be used.
The official ScanCode.io end-to-end tutorial also currently uses ScanCode Workbench 4.0.2 together with:
ignored_dependency_scopes:
- package_type: npm
scope: devDependencies
which makes this project a convenient reproducer.
System configuration
- ScanCode.io: 37.2.0
- ScanCode Toolkit: 32.5.0
- Installation: Docker
- Package ecosystem: npm
- npm lockfile:
package-lock.json, lockfile version 3 - Input: single source code archive containing
scancode-config.yml
I also checked the current ScanCode.io main branch, and ignore_dependency_scope() still appears to use dependency_data.get("package_type") in the same way.
- Lenguaje dominante
- Python
- Estrellas
- 223
- Forks
- 214
- Merge medio
- 2 d 9 min
- PR fusionados (30 d)
- 14
Preparar el entorno
- Incluye un Dockerfile o un archivo de Docker Compose
- Tiene una plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de aboutcode-org/scancode.io
-
Delivery Error in scan_single_package: No connection adapters were found for '/notifications/send_scan_notification/Quizá libre de nuevo @akshat-lakhera la tomó hace 10 días y no hay ningún pull request abierto. Abiertobug
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
aboutcode-org/scancode.io#2206 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
update `run` cli command to run pipelines without an inputPosiblemente ocupada @NoiceHax la tomó hace 75 días. Abiertonice-to-have
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
aboutcode-org/scancode.io#2203 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
Local developement docs are outdatedPosiblemente ocupada @nik-767 la tomó hace 105 días. Abierto
Dificultad 1/5 Menos de una hora Aptitud para principiantes 82/100
aboutcode-org/scancode.io#2197 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
maven-heaven: Update Maven Pipeline's stepsQuizá libre de nuevo @saksham-rathore la tomó hace 7 días y no hay ningún pull request abierto. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 58/100
aboutcode-org/scancode.io#2269 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
[Feature] Multi-row selection and batch triage actions for Resources, Packages, and LicensesQuizá libre de nuevo @saksham-rathore la tomó hace 7 días y no hay ningún pull request abierto. Abierto
Dificultad 5/5 Más de una semana Aptitud para principiantes 48/100
aboutcode-org/scancode.io#2266 · 1 comentario ·
Los mantenedores suelen responder en 1 día
Todos los issues de aboutcode-org/scancode.io
Issues similares
-
changelog investigate
Dificultad 2/5 1-3 horas Aptitud para principiantes 67/100
ramnes/notion-sdk-py#433 ·
-
docs pkg:core
Dificultad 1/5 Menos de una hora Aptitud para principiantes 84/100
pydantic/pydantic-ai#11929 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
intake/intake-esm#789 ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
MemMachine/MemMachine#1877 ·
Los mantenedores suelen responder en 3 días
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
openedx/openedx-authz#579 ·
Los mantenedores suelen responder en 1 día