mfogliatto/ReferenceCop

Add automated end-to-end testing using playground in CI/CD workflows

Open

#43 opened on Dec 20, 2025

 (0 comments) (0 reactions) (0 assignees)C# (2 forks)auto 404
chorehelp wanted

Repository metrics

Stars
 (1 star)
PR merge metrics
 (PR metrics pending)

Description

Overview

Extend the playground project to support automated end-to-end testing that validates ReferenceCop's key scenarios in PR and official build workflows.

Context

Currently, the playground project in playground/ provides a manual testing environment where developers can:

  • Build the ReferenceCop package
  • Test it against sample projects with various rule violations
  • Validate changes end-to-end before committing

This works well for manual validation, but we should automate this to ensure all key scenarios are tested on every check-in.

Goals

  1. Create automated test scenarios that validate all ReferenceCop rule types:

    • AssemblyName rules (e.g., blocking Newtonsoft.Json)
    • ProjectTag rules (e.g., preventing App projects from referencing Tools)
    • ProjectPath rules (e.g., preventing App from referencing Internal libraries)
  2. Integrate these tests into GitHub Actions workflows:

    • Run on every PR build
    • Run on official builds
    • Fail the build if any scenario doesn't produce expected results
  3. Test both positive and negative scenarios:

    • Verify violations are correctly detected (build fails with expected diagnostics)
    • Verify valid references are allowed (build succeeds)

Implementation Approach

Option 1: PowerShell Test Runner

Create a test runner script that:

  • Builds various test configurations in the playground
  • Validates expected build outcomes (success/failure)
  • Checks for expected diagnostic messages
  • Returns appropriate exit codes for CI/CD

Option 2: Integration Test Project

Create a proper MSTest/xUnit test project that:

  • Uses the playground projects as test fixtures
  • Invokes MSBuild programmatically
  • Asserts on build results and diagnostics
  • Integrates with existing test infrastructure

Option 3: Combination

  • Use PowerShell for simple scenario validation
  • Add integration tests for complex validation logic
  • Both run in CI/CD pipeline

Acceptance Criteria

  • Automated tests validate all three rule types (AssemblyName, ProjectTag, ProjectPath)
  • Tests verify both violations (build failure) and valid scenarios (build success)
  • Tests check for expected diagnostic codes (RC0001, RC0002) and messages
  • Tests run in PR build workflow (.github/workflows/pr-build.yaml or similar)
  • Tests run in official build workflow (.github/workflows/official-build.yaml)
  • Failed tests cause the CI/CD build to fail
  • Test results are visible in GitHub Actions UI
  • Documentation updated with automated testing approach

Benefits

  • Catch regressions early in the development cycle
  • Ensure all rule types work correctly across releases
  • Provide confidence when accepting community contributions
  • Validate package integration scenarios automatically
  • Reduce manual testing effort

Related

  • Playground: playground/
  • Test scenarios reference: playground/TEST-SCENARIOS.md
  • Build workflows: .github/workflows/

Contributor guide