Cache refresh policies based on scancode CLI options/scancode output version updates

Open
#5,472 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
15/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Active
Tech stack
python
Domain
tooling

Research direction

The payload names no file or test, so start from the two linked threads: discussion r3948561703 on PR 5289 and issue 3941, which cover cache versioning. The open question is whether to hash the data fields and CLI options into the cache index path, and how stale-cache errors are handled. Maintainers need to settle that design before a patch is meaningful, so done means an agreed policy in the thread, not a code change yet.

Written by the indexing model from the issue text.

Description

maintainers-only

From https://github.com/aboutcode-org/scancode-toolkit/pull/5289#discussion_r3948561703, as a follow up:

Should we include some functionality to re-compute the cache for a file based on data structure changes caused by different versions of scancode, or different CLI options used with different output data formats? There is some discussion on this at https://github.com/aboutcode-org/scancode-toolkit/issues/3941

We need to check that we atleast handle the errors resulting out of this. Additionally it would be nice to somehow include the data format versions in the index cache too, smartly avoiding cache recompute whenever possible.

Output data formats are not always changes based on scancode versions, and an update in the output data format version also doesn't correspond to updates in all the scanner output data. So we need to probably compute a hash based on all the data fields present for a particular scanner data based on the keys present in the mapping and include this in the index file path? This would cover both variations caused by scancode version changes and CLI options used.

Dominant language
Python
Stars
2.7k
Forks
871
Avg merge
1d 10h
Merged PRs (30d)
3

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from aboutcode-org/scancode-toolkit

All issues in aboutcode-org/scancode-toolkit

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.