pySCG: Add a testing framework for Python code examples and documentation

Ouverte
  • Quelque chose ne va pas ?
#7 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub

Personne n'a encore pris cette issue.

  • #1011 par @tommcd — fermée sans fusion

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
20/100
Type d'issue
Fonctionnalité
Clarté
Clairement spécifiée
Activité
À l'abandon
Stack technique
github-actions, python
Domaine
ci-cd, documentation, testing

Piste de recherche

The issue points to docs/Secure-Coding-Guide-for-Python/tests/, pyproject.toml, tox.ini, and .github/workflows/python-tests.yml; begin with tests/README.md and the existing test results in KNOWN_ISSUES.md. The described framework spans code examples, Markdown checks, link validation, and CI across multiple Python versions. Done means those checks and CI behavior match the listed requirements; the issue notes 44 existing problem files for follow-up.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Summary

Add a pytest-based testing framework to validate Python code examples and Markdown documentation structure across the Secure Coding Guide for Python.

[!NOTE]
We moved repo since creating this source code for this solution is on a closed PR in source
https://github.com/ossf/wg-best-practices-os-developers/pull/1011

Motivation

Currently, there is no automated validation of:

  • Python code syntax and deprecation warnings in examples
  • Markdown documentation structure conformance to the template
  • Internal links and code file references
  • Consistency across 100+ CWE articles

This leads to potential issues with broken examples, missing documentation sections, and broken links that could confuse contributors and readers.

Solution

Implement a comprehensive testing framework that validates:

1. Python Code Quality
  • Syntax validation using ast.parse()
  • Deprecation warning detection via subprocess execution
  • Expected output validation against documented examples
  • Support for expected failure markers (intentional issues)
  • Automatic cleanup of test artifacts
2. Markdown Documentation Structure
  • Required sections validation (title, introduction, Non-Compliant, Compliant, Automated Detection, Related Guidelines, Bibliography)
  • Code file reference validation (compliant01.py, noncompliant01.py exist)
  • Table structure validation (Automated Detection, Related Guidelines, Bibliography)
  • Section order validation
3. Link Validation
  • Internal link validation in README.md files
  • Index table link validation
  • Integration with lychee for comprehensive link checking (internal only in CI, external available locally)
4. CI/CD Integration
  • GitHub Actions workflow for automated testing on PRs
  • Smart testing: Only tests changed CWE directories on PRs (fast, ~1 min)
  • Full suite: Runs all tests on pushes to main (~3 min)
  • Multi-version testing (Python 3.9-3.14)
  • Coverage reporting to Codecov
  • Fast execution with uv package manager
5. Local Development Support
  • Easy setup with uv sync --group test
  • Multi-version testing with tox
  • Parallel execution support with pytest-xdist
  • Clean issue reporting with generate_issue_report.py
  • Comprehensive documentation in tests/README.md

Implementation Details

  • Uses pytest with parametrized tests for all Python and Markdown files
  • Organized test utilities in tests/utils/ for maintainability
  • PEP 735 dependency groups for clean dependency management
  • Automatic cleanup of test artifacts (database files, temp files, etc.)
  • Cross-platform Python-based issue report generator

Benefits

  • Ensures all code examples are syntactically valid and run without warnings
  • Validates documentation consistency across all CWE articles
  • Catches broken links and missing files before merge
  • Provides clear, actionable error messages for contributors
  • Reduces review burden by automating quality checks
  • Supports Python 3.9-3.14 compatibility testing
  • Fast CI/CD with smart testing (only changed files on PRs)

Current State

The framework identifies issues in 44 files (documented in KNOWN_ISSUES.md):

  • 31 Documentation files with issues:
    • Missing required sections (Bibliography, Automated Detection, etc.)
    • Missing required tables
    • Section order issues
    • Missing code file references
  • 13 Python code files with issues:
    • 11 files with execution timeouts (intentional infinite loops/blocking operations)
    • 1 file with DeprecationWarning
    • 1 file with output mismatch

These pre-existing issues can be addressed in follow-up PRs. The testing framework will prevent new issues from being introduced.

Files Added/Modified

New Files:

  • .github/workflows/python-tests.yml - CI/CD workflow with smart testing
  • docs/Secure-Coding-Guide-for-Python/tests/ - Complete test framework
  • docs/Secure-Coding-Guide-for-Python/pyproject.toml - Project configuration
  • docs/Secure-Coding-Guide-for-Python/tox.ini - Multi-version testing config
  • docs/Secure-Coding-Guide-for-Python/.gitignore - Ignore test artifacts
  • docs/Secure-Coding-Guide-for-Python/generate_test_summary.sh - Summary script
  • docs/Secure-Coding-Guide-for-Python/KNOWN_ISSUES.md - Reference of existing issues

Modified Files:

  • docs/Secure-Coding-Guide-for-Python/CONTRIBUTING.md - Added testing instructions
  • Fixed typo: Renamed compliants02.py → compliant02.py

Testing

All 765 tests collected, 531 passed, 80 failures (pre-existing issues), 154 skipped.
Tests run successfully across Python 3.9-3.14 in CI/CD.

Langage dominant
Python
Étoiles
2
Forks
2
Merge moyen
11 min
PR mergées (30 j)
2

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de ossf/python-secure-coding-guide

Toutes les issues de ossf/python-secure-coding-guide

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.