Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

Performance issues when running unfurl_version_range_v2advisor (and others)

Ouverte
  • Quelque chose ne va pas ?
#2,333 3 commentaires 1 réaction 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
25/100
Type d'issue
Bug
Clarté
À clarifier
Activité
Calme
Stack technique
postgresql, python
Domaine
backend, databases, performance

Piste de recherche

Le rapport pointe vers le unfurl_version_range_v2advisor improver et les commandes de gestion import --all et improve --all ; commencez par profiler ces points d’entrée en fonction des tailles des tables PostgreSQL et de la requête pg_stat_user_tables fournie. Une contribution nécessiterait une modification des performances définie et reproductible ainsi qu’un critère d’achèvement mesurable, mais l’issue ne précise actuellement ni l’un ni l’autre.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Hi Team,

first release: v34.0.0rc4 (or earlier)
current release: 38.1.0

Our database dates back 2+years and has grown a lot. I checked the sizes of some of the tables using this command:

SELECT
    schemaname AS schema,
    relname AS table_name,
    n_live_tup AS row_count
FROM
    pg_stat_user_tables
ORDER BY
    n_live_tup DESC;

Results were staggering:

"vulnerabilities_impactedpackageaffecting"	735403239 =735 mln (!!!)
"vulnerabilities_advisory"	54630236  = 54 mln
"vulnerabilities_advisoryrelatedalias"	54622255 = 54 mln
"vulnerabilities_packagechangelog"	34332358 =  34 mln
"vulnerabilities_vulnerability_severities"	32511124 =  32 mln
"vulnerabilities_vulnerabilityseverity"	32221132 =  32 mln
"vulnerabilities_impactedpackage"	        25634100 =  25 mln
"vulnerabilities_impactedpackagefixedby"	8558740 =  8mln

We are running VulnerableCode on OpenShift and decided that we wanted a regular updates of the importers and improvers, so we created CronJobs that would import/improve just one thing each (for example python manage.py import nvd_importer_v2). With the new version we have 137 pods running running at different times of day. While this was a lot but sort of manageable (at least we thought) we have starting noticing problems our database was under enormous load especially when improvers run. While I know that this is caused by parallel runs there is one specific improver that causes our nodes to crash and push the database beyond CPU limits - unfurl_version_range_v2advisor it was running usually for 4 days straight and using north of 44 GiB of RAM on OpenShift and 10-12 CPUs on PostgreSQL side.

With this we tried running importers and improvers as it says in documentation import --all and improve --all but those takes days or weeks to do.

Most recent importer --all run:

CommandError: 8 failed!: apache_kafka_importer_v2,mattermost_importer_v2,gitlab_importer,pypa_importer,npm_importer,nginx_importer,pysec_importer,vulnerabilities.importers.istio.IstioImporter

and it still took 26 hrs. and that is just importers on their own.

We haven't been able to do full run of the improve -all as of yet as I have seen some of those improvers can run for 3-4 days each.

So the question here arises, what to do with this as with this version it is unusable for our production instances.

what to do?

  1. Delete the whole thing and start over with new db
  2. Run importers once every 26 hrs, and then start improve --all which will take most likely weeks
  3. Skip most resource intensive improvers and just do the rest?

I would love to hear your suggestions on possible solutions, or how you run it?

Langage dominant
Python
Étoiles
716
Forks
347
Merge moyen
12 h 15 min
PR mergées (30 j)
9

Préparer son environnement

  • Fournit un Dockerfile ou un fichier Docker Compose
  • Aucun modèle de pull request
  • Aucun guide de contribution

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de aboutcode-org/vulnerablecode

Toutes les issues de aboutcode-org/vulnerablecode

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.