promtool support for coverage of rules

Proposal

We are using GitOps to write, validate and deploy Prometheus rules.

Currently, promtool check config/rules only writes to standard output and reports failure with return code. This is enough for deployment gating depending on the success/failure of the checks.

Concerning check of rules, only the code review can catch the case of a rule added without relevant test. Being humans in a busy world, we can "miss" a rule and deploy the configuration without relevant test.

Would it be possible to have an output, understandable by a script, preferably using a well known format which would:

• report individual test success/error (that's for the nice display in Jenkins by example)
• report coverage of rules in a file (much harder)

I recognize that the coverage request is opening a (small) can of worm but it is the most interesting one IMHO.

Note: I think a minimal xunit/junit format would be a good option since it is widely supported

Example:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
    <!-- test suite could correspond to test_group  -->
    <testsuite timestamp="2023-01-13T10:05:06">
         <!-- test_group name for "className"  -->
         <!-- for name of case, nothing exists ; index in suite may be enough - for alert_test_case, alertame coube part of it. -->
        <testcase name="???" classname="test_group.name" time="0.001">
            <failure message="FAILED alertname:..." type="result.DoesntMatch">
                <!-- message -->
                exp:"[....]"
                got:"[...]"
            </failure>
        </testcase>
    </testsuite>
</testsuites>

Coverage would be more tricky. For my use case, I would settle for a flag generating errors is a rule hasn't been tested in a run (could be an informational entry in output).