Testing: mutation testing
Line coverage records that a line ran. It cannot show whether any test checked the result. This course applies mutation testing to one Rails repository with 1,206 examples and 82.99% line coverage. It covers the mutation operators and a hand-run experiment of eleven mutations, with six killed and five surviving. It also covers equivalent mutants, the cost of triage and the mutant tool for Ruby. It closes by writing boundary tests for the password policy and giving each survivor a verdict.
What you'll learn
- ✓Why line coverage cannot tell an executed line from a tested one
- ✓The mutation operator families and what each surviving family indicates
- ✓How to tell equivalent mutants apart from missing tests
- ✓The three verdicts for a survivor: write the test, delete the code, or accept it
- ✓Running mutant on a single class, plus the shell loop that needs no gem
- ✓Writing boundary pairs that assert on behavior instead of shape
Coverage says the line ran
This repository's test suite reports 82.99% line coverage over 1,206 examples. That is a respectable number, and this course is about what even a perfect one would not tell you.
Read the full lesson free →Curriculum
7 lessonsCoverage says the line ran
Line coverage shows a line executed, and a weakened password minimum still leaves all 1,206 examples green.
The mutation operators
The mutation families, from boundaries to call removals, plus the vocabulary of killed, alive, timeout, noop and equivalent.
The experiment, run on this repository
Eleven hand-applied mutations on this repository: six killed, five survived, three of them in the password policy.
Equivalent mutants and the cost of triage
Equivalent mutants, the three verdicts for each survivor, and why triage cost calls for scoping runs to one class.
Running mutant
Running the mutant gem on one named subject, its licensing modes, and the shell loop that needs no gem.
Killing the survivors
Seven boundary examples kill the password policy survivors, while the unreachable CommentPolicy guard gets a delete verdict.
Putting it together: the question behind it
Mutation testing as one question, where it helps, where it does not, and the testing habits worth keeping.