pySCG: Add a testing framework for Python code examples and documentation

Abierto
  • ¿Algo no está bien?
#7 1 comentario 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

  • #1011 de @tommcd — cerrado sin fusionar

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
20/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Estancado
Stack tecnológico
github-actions, python
Área
ci-cd, documentation, testing

Línea de trabajo

The issue points to docs/Secure-Coding-Guide-for-Python/tests/, pyproject.toml, tox.ini, and .github/workflows/python-tests.yml; begin with tests/README.md and the existing test results in KNOWN_ISSUES.md. The described framework spans code examples, Markdown checks, link validation, and CI across multiple Python versions. Done means those checks and CI behavior match the listed requirements; the issue notes 44 existing problem files for follow-up.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

Add a pytest-based testing framework to validate Python code examples and Markdown documentation structure across the Secure Coding Guide for Python.

[!NOTE]
We moved repo since creating this source code for this solution is on a closed PR in source
https://github.com/ossf/wg-best-practices-os-developers/pull/1011

Motivation

Currently, there is no automated validation of:

  • Python code syntax and deprecation warnings in examples
  • Markdown documentation structure conformance to the template
  • Internal links and code file references
  • Consistency across 100+ CWE articles

This leads to potential issues with broken examples, missing documentation sections, and broken links that could confuse contributors and readers.

Solution

Implement a comprehensive testing framework that validates:

1. Python Code Quality
  • Syntax validation using ast.parse()
  • Deprecation warning detection via subprocess execution
  • Expected output validation against documented examples
  • Support for expected failure markers (intentional issues)
  • Automatic cleanup of test artifacts
2. Markdown Documentation Structure
  • Required sections validation (title, introduction, Non-Compliant, Compliant, Automated Detection, Related Guidelines, Bibliography)
  • Code file reference validation (compliant01.py, noncompliant01.py exist)
  • Table structure validation (Automated Detection, Related Guidelines, Bibliography)
  • Section order validation
3. Link Validation
  • Internal link validation in README.md files
  • Index table link validation
  • Integration with lychee for comprehensive link checking (internal only in CI, external available locally)
4. CI/CD Integration
  • GitHub Actions workflow for automated testing on PRs
  • Smart testing: Only tests changed CWE directories on PRs (fast, ~1 min)
  • Full suite: Runs all tests on pushes to main (~3 min)
  • Multi-version testing (Python 3.9-3.14)
  • Coverage reporting to Codecov
  • Fast execution with uv package manager
5. Local Development Support
  • Easy setup with uv sync --group test
  • Multi-version testing with tox
  • Parallel execution support with pytest-xdist
  • Clean issue reporting with generate_issue_report.py
  • Comprehensive documentation in tests/README.md

Implementation Details

  • Uses pytest with parametrized tests for all Python and Markdown files
  • Organized test utilities in tests/utils/ for maintainability
  • PEP 735 dependency groups for clean dependency management
  • Automatic cleanup of test artifacts (database files, temp files, etc.)
  • Cross-platform Python-based issue report generator

Benefits

  • Ensures all code examples are syntactically valid and run without warnings
  • Validates documentation consistency across all CWE articles
  • Catches broken links and missing files before merge
  • Provides clear, actionable error messages for contributors
  • Reduces review burden by automating quality checks
  • Supports Python 3.9-3.14 compatibility testing
  • Fast CI/CD with smart testing (only changed files on PRs)

Current State

The framework identifies issues in 44 files (documented in KNOWN_ISSUES.md):

  • 31 Documentation files with issues:
    • Missing required sections (Bibliography, Automated Detection, etc.)
    • Missing required tables
    • Section order issues
    • Missing code file references
  • 13 Python code files with issues:
    • 11 files with execution timeouts (intentional infinite loops/blocking operations)
    • 1 file with DeprecationWarning
    • 1 file with output mismatch

These pre-existing issues can be addressed in follow-up PRs. The testing framework will prevent new issues from being introduced.

Files Added/Modified

New Files:

  • .github/workflows/python-tests.yml - CI/CD workflow with smart testing
  • docs/Secure-Coding-Guide-for-Python/tests/ - Complete test framework
  • docs/Secure-Coding-Guide-for-Python/pyproject.toml - Project configuration
  • docs/Secure-Coding-Guide-for-Python/tox.ini - Multi-version testing config
  • docs/Secure-Coding-Guide-for-Python/.gitignore - Ignore test artifacts
  • docs/Secure-Coding-Guide-for-Python/generate_test_summary.sh - Summary script
  • docs/Secure-Coding-Guide-for-Python/KNOWN_ISSUES.md - Reference of existing issues

Modified Files:

  • docs/Secure-Coding-Guide-for-Python/CONTRIBUTING.md - Added testing instructions
  • Fixed typo: Renamed compliants02.py → compliant02.py

Testing

All 765 tests collected, 531 passed, 80 failures (pre-existing issues), 154 skipped.
Tests run successfully across Python 3.9-3.14 in CI/CD.

Lenguaje dominante
Python
Estrellas
2
Forks
2
Merge medio
11 min
PR fusionados (30 d)
2

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de ossf/python-secure-coding-guide

Todos los issues de ossf/python-secure-coding-guide

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.