C# custom executable test runners are classified as no_tests with no quality limitation

Open
  • Is something wrong?
#892 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
1-2 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
csharp, python
Domain
testing-qa, tooling

Research direction

Start with desloppify/languages/csharp/test_coverage.py, then trace how analyze_test_quality in desloppify/engine/detectors/coverage/mapping_analysis.py handles unrecognized functions and how quality_issue_item in desloppify/engine/detectors/test_coverage/_issue_quality.py reports them. Run the pure detector APIs from the issue’s reproduction against the synthetic C# harness. Done means the harness is recognized with supported evidence or its test quality is reported as unknown, without awarding automatic thoroughness credit.

Written by the indexing model from the issue text.

Description

A small C# executable test harness is discovered as test-zone source and maps to the production code, but its scenario functions and throwing assertion helper are classified as no_tests. This leaves test quality unknown without surfacing that limitation.

Reproduced with Desloppify 1.0 plus #890 at 0bba7a445d7164b0114bf425d2cb03cff5196b2c, Python 3.14, the default C# graph fallback, and no Roslyn command. All source below is synthetic. The quality patterns are unchanged by #890.

Minimal fixture

Create these files below a repro/ directory. A single SDK project targeting net10.0 with OutputType=Exe can contain them; no external test framework is used.

// src/Counter.cs
namespace Sample;
public sealed class Counter
{
    public int Value { get; private set; }
    public void Add(int amount)
    {
        if (amount < 0)
            throw new System.ArgumentOutOfRangeException(nameof(amount));
        Value += amount;
    }
}

// tests/Check.cs
namespace Sample.Tests;
public static class Check
{
    public static void That(bool condition, string message)
    {
        if (!condition)
            throw new System.Exception(message);
    }
}

// tests/CounterScenarios.cs
using Sample;
namespace Sample.Tests;
public static class CounterScenarios
{
    public static void AddsPositiveAmounts()
    {
        var counter = new Counter();
        counter.Add(2);
        Check.That(counter.Value == 2, "amount is retained");
        counter.Add(3);
        Check.That(counter.Value == 5, "amounts accumulate");
    }
}

// tests/Program.cs
using System;
using Sample.Tests;
var scenarios = new (string Name, Action Body)[]
{
    ("positive amounts accumulate", CounterScenarios.AddsPositiveAmounts)
};
var failed = 0;
foreach (var (name, body) in scenarios)
{
    try { body(); Console.WriteLine($"PASS {name}"); }
    catch (Exception error) { failed++; Console.Error.WriteLine(error); }
}
return failed == 0 ? 0 : 1;

From repro/, with Desloppify importable, this runs the pure detector APIs rather than a scan and writes no scanner state:

from pathlib import Path
from desloppify.engine.detectors.coverage.mapping import analyze_test_quality
from desloppify.engine.detectors.test_coverage.detector import detect_test_coverage
from desloppify.engine.policy.zones import COMMON_ZONE_RULES, FileZoneMap
from desloppify.languages.csharp.detectors.deps import build_dep_graph

files = sorted(str(p) for p in Path(".").rglob("*.cs"))
tests = {p for p in files if p.startswith("tests/")}
print(analyze_test_quality(tests, "csharp"))
print(detect_test_coverage(
    build_dep_graph(Path.cwd()), FileZoneMap(files, COMMON_ZONE_RULES), "csharp"
)[0])
Actual pure detector result

For tests/CounterScenarios.cs:

{
  "assertions": 0,
  "mocks": 0,
  "test_functions": 0,
  "snapshots": 0,
  "placeholder": false,
  "quality": "no_tests"
}

The helper and runner receive the same quality classification. The graph correctly has CounterScenarios.cs -> Counter.cs, and detect_test_coverage(...)[0] is [].

This is not a claim that test-file discovery fails, or that this quality classification causes every coverage gap. Test files are selected by zone, independently of attribute recognition. The narrower problem is that an executable, failure-propagating scenario harness has no usable quality analysis, while the result says no_tests and produces no quality limitation.

Expected / design question

Either provide a supported, generalizable way to identify registered scenario delegates and assertion helpers, or explicitly report unsupported/unknown test quality. This example has one registered scenario and two calls to a helper that throws on a false condition; treating those as confidently absent is misleading.

Counting every public static method as a test, or adding a literal Check.That/Test.True regex, would overclaim evidence for arbitrary projects. A language-specific analysis hook or explicit runner/assertion contract may be more appropriate. The expected outcome is honest recognition or uncertainty, not automatic "thorough" credit.

Relevant paths:

  • desloppify/languages/csharp/test_coverage.py: TEST_FUNCTION_RE recognizes attribute-only tests; ASSERT_PATTERNS recognizes specific framework spellings.
  • desloppify/engine/detectors/coverage/mapping_analysis.py::analyze_test_quality: zero recognized functions becomes no_tests.
  • desloppify/engine/detectors/test_coverage/_issue_quality.py::quality_issue_item: no_tests has no emitted quality item.

I checked existing C# coverage issues and #890 before filing. #52 motivates test-depth analysis generally; this report concerns unsupported custom executable harness semantics. No project-specific references, namespace-wide coverage expansion, source changes, or score adjustments are proposed.

Dominant language
Python
Stars
3.8k
Forks
301
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from peteromallet/desloppify

All issues in peteromallet/desloppify

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.