match_unmapped_resources ignores matched_extensions and re-sends those files to PurlDB
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Newbie friendliness
- 25/100
Research direction
Read match_unmapped_resources() in scanpipe/pipes/d2d.py and the corresponding tests in test_d2d.py. The issue reports that the result of .exclude() is discarded; confirm which resources reach the mocked PurlDB matcher when matched_extensions=[".java"]. Done means only the unmatched extension reaches the matcher and the test passes; a linked pull request is already open.
Written by the indexing model from the issue text.
Description
Describe the bug
match_unmapped_resources() in scanpipe/pipes/d2d.py is supposed to skip the extensions that were already looked up in PurlDB by the match_resources_to_purldb step (matched_extensions=self.ecosystem_config.matchable_resource_extensions), but the filter is never applied:
if matched_extensions:
to_unmapped.exclude(extension__in=matched_extensions)
The result of .exclude() is discarded, so to_unmapped still contains every file with a matched extension (for example all the .class files of a Java project) and they are sent to PurlDB a second time.
System configuration
- ScanCode.io
main(39acd55) - Local install, macOS
- Pipeline:
map_deploy_to_develop
To Reproduce
- Create a project with two
to/files that have no status, for exampleBaz.javaandexpected.json. - Call
d2d.match_unmapped_resources(project, matched_extensions=[".java"], logger=print)with_match_purldb_resourcesmocked. - The log says
Mapping 2 to/ resources with empty status in PurlDB using SHA1andBaz.javais passed to the PurlDB matcher.
Expected behavior
Only expected.json is sent to PurlDB (Mapping 1 to/ resources ...). Files with a matched extension still get the requires-review status at the end of the step, as they do today.
Proposed fix: assign the result, to_unmapped = to_unmapped.exclude(extension__in=matched_extensions), plus a unit test in test_d2d.py that calls match_unmapped_resources with matched_extensions and checks which resources reach the matcher. I have the change and the test ready locally and can open a PR.
- Dominant language
- Python
- Stars
- 226
- Forks
- 222
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 32
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from aboutcode-org/scancode.io
-
Register Missing Models in Django Admin (DiscoveredLicense, ProjectMessage, CodebaseRelation, Run, WebhookSubscription)Possibly taken @Rucha0901 claimed this 5 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
aboutcode-org/scancode.io#2329 ·
Maintainers usually reply within 1 day
-
ignored_dependency_scopes does not exclude npm devDependenciesPossibly taken @MGpromax claimed this 83 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
aboutcode-org/scancode.io#2215 ·
Maintainers usually reply within 1 day
-
Delivery Error in scan_single_package: No connection adapters were found for '/notifications/send_scan_notification/May be free again @akshat-lakhera claimed this 30 days ago, and no pull request is open. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
aboutcode-org/scancode.io#2206 · 1 comment ·
Maintainers usually reply within 1 day
-
update `run` cli command to run pipelines without an inputPossibly taken @NoiceHax claimed this 95 days ago. Opennice-to-have
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
aboutcode-org/scancode.io#2203 · 1 comment ·
Maintainers usually reply within 1 day
-
Local developement docs are outdatedPossibly taken @nik-767 claimed this 125 days ago. Open
Difficulty 1/5 Under an hour Newbie friendliness 82/100
aboutcode-org/scancode.io#2197 · 1 comment ·
Maintainers usually reply within 1 day
All issues in aboutcode-org/scancode.io
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
GoogleCloudPlatform/professional-services-data-validator#1899 ·
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
mohanraj00/entelos#200 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 68/100
-
sweep
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
bketelsen/truenas-mcp#77 ·
Maintainers usually reply within 1 day
-
Console writes URL credentials to the local log, though AGENTS.md says secrets never reach diskOpen
Difficulty 1/5 Under an hour Newbie friendliness 86/100
sandbox-quantum/switch#2163 ·
Maintainers usually reply within 1 day