How test cases can make you blind to defects

New Thinking 18 November 2013 Nicola Owen

Inattentional blindness

When you write a test script before looking at the system, you write it with a specific goal in mind. Maybe you want to make sure that the Search box is working as intended or you are checking the default values in all of the fields in a registration screen. This approach can make software testing seem a bit more black and white than it actually is as it potentially encourages a ‘checkbox’ approach to testing. As a result, you risk missing what’s right in front of you because you’re too busy focusing on something else.

Let me paint you a picture…

You’ve just received the latest build into the test environment and have just started testing.

Your mission: To run a test that checks that a registered user can successfully log in to the system.

You run this test and it passes. “Success!” you might think. But then you didn’t try clicking ‘Cancel’. Little do you know, but clicking ‘Cancel’ after you have entered you user name but not your password, results in all of the fields being cleared without asking for confirmation. Now, you might have been planning to run a few tests to check the effect of clicking ‘Cancel’… but that isn’t a given.

The moon-walking bear

I’d almost liken this situation to the moon walking bear that I first came across at a Management 101 lecture in my first year at uni. Watch the video here.

At the beginning of the video, you’re asked, “How many passes does the team in white make?”, followed by a series of passes between the players in the white team and those in the black team. You watch closely in order to answer this question correctly. Lo and behold, chances are you do.

Then you are asked, “Did you see the moon-walking bear?”

What moon-walking bear?!

Most people watching the video for the first time miss it. You see, you were so busy focusing on the task at hand that you were blind to what was right in front of you. Only after the watching the video again are you truly convinced that it was there all along.

Jot it down

Sticking to test cases and not deviating from them to learn more about the system has the potential to limit your vision by ensuring you focus on what you are specifically looking for. Although there is the benefit of ensuring you spot a defect that is the target of your test case, there is also the real risk of missing something not covered by your current test case. 

It is important to be aware of this common and potentially detrimental flaw of sticking to test cases and not deviating from them. Noticed something strange that isn’t part of your test case? Or maybe you just came up with a test case idea while you’re executing. Jot it down on a Post-it note and stick it up on the edge of your screen.

A defect might have been right in front of you, but you were too busy focusing on something else. And we don’t want that to happen.