You may have heard the phrase ‘Insanity is doing the same thing over and over again and expecting different results’, and by that definition testing can be insane. (I’m sure there are many testers out there who would agree!)

Tests, are of course, meant to be run repeatedly. We want to prevent regressions introduced by new changes to code, and one way to do this is to re-run your test suite and check nothing has changed. This is not insane, but sensible.

Where we run into insanity is where we expect these same, repeated tests to continually pick up new defects. When first written and executed, these tests may well have identified new bugs within the code. But once those bugs are fixed, re-running the tests again and again and expecting other new bugs to found every time could be considered insane.

This insanity is also known as the Pesticide Paradox, and avoiding it is one of the seven principles of testing.

The Pesticide Paradox states that if the same tests are repeated over and over again, the same test cases will eventually stop identifying new bugs in the system, like how pests become immune to the same pesticides over time.

It’s obvious then, that if we want to be assured that our software or system is thoroughly tested, that we need to avoid this paradox. If we don’t we can be lulled into a false sense of security, assuming that finding no new bugs means our code is bug-free. Here, it is important to remember another of the seven testing principles: testing shows the presence, not absence, of bugs.

In an age where test automation is on the rise, succumbing to the Pesticide Paradox is relatively easy, especially if you don’t have a dedicated testing team with the expertise to help you navigate around it. This does not mean that automated tests should be abandoned entirely, as they do provide value in saving time and preventing regression, but rather that dynamic testing activities need to be done alongside them.

Dynamic testing activities that can help you avoid the Pesticide Paradox include:

• Exploratory Testing. Mike Harris wrote a fantastic blog on this. He says that “exploratory testing involves scouting around the areas that tests do not cover”, and so by doing this we can cover more test scenarios and identify or rule out more defects.
• Involve different testers. Ensuring your testing team is diverse means that you will get new perspectives on your code. Involve testers, developers, domain experts – anyone who might spot a new approach.
• Regularly review your test suite. Are your tests still fir for purpose? Has the code base evolved much since those tests are written? Are there new gaps to be covered? Are some tests now redundant?
• Modify your test data. As well as writing new tests, changing the data inputs can also help keep tests fresh and able to spot new defects. This can either be done strategically using techniques like boundary value analysis, or you can use a constrained random input generation method to help get a broader range of data.