Tests Deadliness and Higher Order Mutations

オープン
  • 何か問題がありますか？
#174 コメント 2 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
15/100
issue の種類
機能追加
明瞭さ
説明が足りない
活発さ
停滞
技術スタック
python
領域
cli, testing-qa

調査の方向性

まず、既存の mutmut results の動作と CLI オプションの処理を読んでから、issue 113 の関連する議論を確認してください。issue にはファイルやテストの指定がありません。完了とするには、テストの致死性と上限付き高次ミューテーション（--max_order を含む）について、範囲を決定し、動作を文書化する必要があります。

索引モデルが issue の本文から書いたものです。

説明

Hey boxed and mutmut contributors,
thanks for this great library! It really helps in developing tests, which test the real corner cases!

I was wondering, whether it would be possible to implement two new features, namely test deadliness and higher order mutations.

Test Deadliness

As far as I am aware, mutmut does not report the number of tests which killed a mutant. The more tests fail after applying a mutant, the higher the deadliness for said mutant is. I think this would be interesting, as this reveals which locations are really critical for the correctness of the code base. It also reveals which locations are well tested and which are not.
As an example, a surviving mutant has a test deadliness of 0, a mutant killed by a single failed test has a test deadliness of 1 and so on.
Maybe this could be normalized by the number of tests to account for small or large test bases.

Reporting an aggregate measure (average, median) or the histogram of test deadliness over all mutants reveals whether

  • there are too few tests (aggregate test deadliness close to 0),
  • there are just a few independent tests, which address individual lines (aggregate test deadliness around 1) or
  • there are also very sensitive tests, which test corner cases of higher level functions (aggregate test deadliness above 1).

As this should be quite easy to obtain I would propose just outputting the results when calling mutmut results.

Higher Order Mutations

Mutmut currently only applies a single mutation before running tests. Combining these mutations (two mutations -> second order, three mutations -> third order, an so on) would allow to evaluate the test robustness in a more extreme setting (edit: I noticed, that you mentioned this idea already in https://github.com/boxed/mutmut/issues/113#issuecomment-484938645). I would argue that this setting (for reasonable low orders) is still quite realistic, as the example below shows:

  • for i in range(0,5): ... the reference code
  • for i in range(1,5): ... a normal first order mutant, with changed iteration start and thus the iteration length
  • for i in range(0,6): ... a normal first order mutant, with changed iteration end and thus the iteration length
  • for i in range(1,6): ... a second order mutant, with changed iteration start and end, but not the iteration length

Another example would be an if A and B or (C and not D): ... statement with multiple logical expressions A, B, C and D. A single mutant will likely change the behavior, but multiple mutations could partially restore parts of the logical behavior and thus have tests wrongly pass.

Obviously this does not scale well for high orders if the number of mutants N is large, I think there are potentially N*(N-1)/2 higher order mutants, which would need to be tested. But within all potential mutant combinations there are some, which are more interesting and more likely to survive. Those combinations are most likely those, of which the independent mutant weren't able to kill (m)any tests.
Thus, building on the test deadliness proposed above, one could order or limit the higher order mutations by the test deadliness of their individual tests. Afterall two (or more) mutations, which were not killed, might also survive in conjunction.

Note, that test deadliness should also be measured for these higher order mutants, which allows us to do the following:
For example, if mutations A and B have td=0, testing AB would be interesting. On the other hand if mutations A and B have td>>0, testing AB would not be interesting.
If if mutations A ,B and C have td=0, the second order mutations AB, AC and CB can be tested. Furthermore the third order mutation ABC could be tested, and this would be determined by the test deadliness of he second order mutations AB, AC and CB.

To use this feature, I would propose a new CLI argument --max_order, which defaults to 1 and thus defaults to the current mutmut behavior. A usage like --max_order=2 would additionally test all possible pair combinations of the mutants. Low digit values of --max_order should reveal the most interesting combinations, as more and more concurrent mutations increase the probability of a test or simply a execution failure.
Something like --td_limit_higher_order could be used to provide a test deadliness limit to skip mutant combinations, where the individuals already have a high average test deadliness.

I would be happy to hear your or any of the mutmut contributors opinion on the usefulness of these two features!

主要言語
Python
スター
1.5k
フォーク
190
平均マージ
1日 11時間
マージ済み PR（30日）
8

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

boxed/mutmut のほかの issue

boxed/mutmut の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。