pySCG: Add a testing framework for Python code examples and documentation

Đang mở
  • Có gì không ổn?
#7 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Chưa có ai nhận issue này.

  • #1011 của @tommcd — đã đóng, không merge

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
20/100
Loại issue
Tính năng
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Đình trệ
Công nghệ
github-actions, python
Lĩnh vực
ci-cd, documentation, testing

Hướng nghiên cứu

The issue points to docs/Secure-Coding-Guide-for-Python/tests/, pyproject.toml, tox.ini, and .github/workflows/python-tests.yml; begin with tests/README.md and the existing test results in KNOWN_ISSUES.md. The described framework spans code examples, Markdown checks, link validation, and CI across multiple Python versions. Done means those checks and CI behavior match the listed requirements; the issue notes 44 existing problem files for follow-up.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Summary

Add a pytest-based testing framework to validate Python code examples and Markdown documentation structure across the Secure Coding Guide for Python.

[!NOTE]
We moved repo since creating this source code for this solution is on a closed PR in source
https://github.com/ossf/wg-best-practices-os-developers/pull/1011

Motivation

Currently, there is no automated validation of:

  • Python code syntax and deprecation warnings in examples
  • Markdown documentation structure conformance to the template
  • Internal links and code file references
  • Consistency across 100+ CWE articles

This leads to potential issues with broken examples, missing documentation sections, and broken links that could confuse contributors and readers.

Solution

Implement a comprehensive testing framework that validates:

1. Python Code Quality
  • Syntax validation using ast.parse()
  • Deprecation warning detection via subprocess execution
  • Expected output validation against documented examples
  • Support for expected failure markers (intentional issues)
  • Automatic cleanup of test artifacts
2. Markdown Documentation Structure
  • Required sections validation (title, introduction, Non-Compliant, Compliant, Automated Detection, Related Guidelines, Bibliography)
  • Code file reference validation (compliant01.py, noncompliant01.py exist)
  • Table structure validation (Automated Detection, Related Guidelines, Bibliography)
  • Section order validation
3. Link Validation
  • Internal link validation in README.md files
  • Index table link validation
  • Integration with lychee for comprehensive link checking (internal only in CI, external available locally)
4. CI/CD Integration
  • GitHub Actions workflow for automated testing on PRs
  • Smart testing: Only tests changed CWE directories on PRs (fast, ~1 min)
  • Full suite: Runs all tests on pushes to main (~3 min)
  • Multi-version testing (Python 3.9-3.14)
  • Coverage reporting to Codecov
  • Fast execution with uv package manager
5. Local Development Support
  • Easy setup with uv sync --group test
  • Multi-version testing with tox
  • Parallel execution support with pytest-xdist
  • Clean issue reporting with generate_issue_report.py
  • Comprehensive documentation in tests/README.md

Implementation Details

  • Uses pytest with parametrized tests for all Python and Markdown files
  • Organized test utilities in tests/utils/ for maintainability
  • PEP 735 dependency groups for clean dependency management
  • Automatic cleanup of test artifacts (database files, temp files, etc.)
  • Cross-platform Python-based issue report generator

Benefits

  • Ensures all code examples are syntactically valid and run without warnings
  • Validates documentation consistency across all CWE articles
  • Catches broken links and missing files before merge
  • Provides clear, actionable error messages for contributors
  • Reduces review burden by automating quality checks
  • Supports Python 3.9-3.14 compatibility testing
  • Fast CI/CD with smart testing (only changed files on PRs)

Current State

The framework identifies issues in 44 files (documented in KNOWN_ISSUES.md):

  • 31 Documentation files with issues:
    • Missing required sections (Bibliography, Automated Detection, etc.)
    • Missing required tables
    • Section order issues
    • Missing code file references
  • 13 Python code files with issues:
    • 11 files with execution timeouts (intentional infinite loops/blocking operations)
    • 1 file with DeprecationWarning
    • 1 file with output mismatch

These pre-existing issues can be addressed in follow-up PRs. The testing framework will prevent new issues from being introduced.

Files Added/Modified

New Files:

  • .github/workflows/python-tests.yml - CI/CD workflow with smart testing
  • docs/Secure-Coding-Guide-for-Python/tests/ - Complete test framework
  • docs/Secure-Coding-Guide-for-Python/pyproject.toml - Project configuration
  • docs/Secure-Coding-Guide-for-Python/tox.ini - Multi-version testing config
  • docs/Secure-Coding-Guide-for-Python/.gitignore - Ignore test artifacts
  • docs/Secure-Coding-Guide-for-Python/generate_test_summary.sh - Summary script
  • docs/Secure-Coding-Guide-for-Python/KNOWN_ISSUES.md - Reference of existing issues

Modified Files:

  • docs/Secure-Coding-Guide-for-Python/CONTRIBUTING.md - Added testing instructions
  • Fixed typo: Renamed compliants02.py → compliant02.py

Testing

All 765 tests collected, 531 passed, 80 failures (pre-existing issues), 154 skipped.
Tests run successfully across Python 3.9-3.14 in CI/CD.

Ngôn ngữ chính
Python
Star
2
Fork
2
Merge trung bình
11 phút
Pull request đã merge (30 ngày)
2

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của ossf/python-secure-coding-guide

Tất cả issue của ossf/python-secure-coding-guide

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.