Adjust coverage resolution based on the number of tests · hedgehogqa/haskell-hedgehog#309

(0 comments) (2 reactions) (0 assignees)Haskell (373 stars) (52 forks)batch import

bughelp wanted

Description

When running more than 100 tests, the resolution of the coverage output can be confusing:

✗ prop_validates_age failed
   after 1000 tests.
   exactly 18, validated on common day 13% ██▌················· ✓  5%
   exactly 18, validated on leap day    5% █··················· ✓  1%
   close to 18, non-leapling           27% █████▎·············· ✓  5%
 ⚠ close to 18, leapling                1% ···················· ✗  1%
   too young                           30% █████▉·············· ✓ 10%
   not yet born                         5% ▉··················· ✓  1%
   old enough                          70% ██████████████······ ✓ 30%

In this case: ⚠ close to 18, leapling 1% ···················· ✗ 1%

We only covered 0.9% but it is rounded to 1% when displayed.

We should adjust the precision of displayed coverage based on the number of tests, so something like:

-- 100 tests
   close to 18, non-leapling           27% █████▎·············· ✓  5%
 ⚠ close to 18, leapling                0% ···················· ✗  1%

-- 1000 tests
   close to 18, non-leapling         27.4% █████▎·············· ✓  5%
 ⚠ close to 18, leapling              0.9% ···················· ✗  1%

-- 10000 tests
   close to 18, non-leapling        27.39% █████▎·············· ✓  5%
 ⚠ close to 18, leapling             0.09% ···················· ✗  1%

-- 100000 tests
   close to 18, non-leapling       27.392% █████▎·············· ✓  5%
 ⚠ close to 18, leapling            0.009% ···················· ✗  1%

-- etc

Coverage requirements (the right column) can only be integers, at least at the moment. So we might as well keep them with no decimal places.

I suspect this might need to be changed so there is no rounding. It is probably a hangover from before the rounding was done in the reporting code itself.

Contributor guide

Tech stack: None
Domain: testing
Issue type: feature
Difficulty: 3
Estimated time: 1-3 hours
Activity status: stale
Clarity: clear
Prerequisites: Haskell programmingHedgehog test frameworkProperty based testing
Newbie friendliness: 70
Research direction: Start by examining the file `hedgehog/src/Hedgehog/Internal/Property.hs`, specifically lines 999-1010 where coverage rounding currently occurs. The issue asks to remove the rounding and instead adjust the number of decimal places displayed based on the total number of tests (e.g., 1 decimal for 1000 tests, 2 for 10000, etc.). Note that coverage requirements in the right column remain integers. Consider how to compute the desired precision dynamically, and update the formatting logic accordingly. Also ensure that the change doesn't break existing tests.