Watch out for the fishhooks! Diminishing returns, AI and Test Analysis

By now, you will have heard of the emergence of AI tools and perhaps experimented with them yourself (who are we kidding!) With Test Analysis tasks an obvious candidate for AI assistance, several questions are begged: how are we using it for our daily work? Where and how does it add value? Are there limitations to what AI can do? And are there fishhooks to watch out for?

Let’s get to it.

Firstly, yes, AI is hugely valuable to Test Analysis. Secondly, there are distinct limitations which expressly require the oversight and expertise of a trained professional.

That’s another way of saying AI isn’t replacing skilled testers, at least not any time soon, and probably not any time in the future either. The trick, therefore, is that as with any other tool, we need to use AI where it adds value while recognising the point where that value rapidly diminishes.

And the real key here is it takes a competent, skilled tester to recognise where AI helps, where it hinders, and the point where the accelerator becomes the brake pedal.

The diminishing returns are something like a fishhook on its side: a J-curve, where there’s rapid value creation, until the hook curves away and there is rapid value erosion. And the pointy bits? Those are the entirely new problems caused by AI, like hallucination and factual errors.

Notes from the field

Let’s take a real-world example with a typical test analyst task. Along with a colleague, I recently put AI to work generating test cases for a set of user stories; both of us are seasoned QA professionals and reasonably skilled at AI prompting.

The job involved writing test cases for user stories which included boundary conditions. We spent the best part of a day trying to get useable test cases for a user story by using larger and larger models, while refining output along the way.

Finally, we reached a point of confidence in the expected test results and could hand over to a tester and say, ‘here you go, test this’.

Could the AI do the job? Yes. Was the AI useful? Yes, to a point. Did it save me time? No.   

Why? Because we spent many hours tweaking and fiddling the AI. And that took longer than simply doing the work the ‘traditional’ way. It was a case of rapidly diminishing returns.

Don’t abandon the good with the bad

We’re exploring AI in testing because it can and does add value. It also can and does subtract value if its capabilities and limitations aren’t known.

What we found through this exercise (and many others) is that yes, the AI got things going very quickly, establishing the basis for our test cases in minutes. But as we discovered, as it hit that J hook it just as quickly stopped being useful and started slowing things down.

In other words, recognising when the AI has provided a good enough starting point is an essential skill. And knowing when that J curve kicks in means making the best use of AI as a tool accelerating performance.

AI’s allure therefore comes with caveats, the biggest of which is that you need to be a skilled and experienced person to make the best use of it.

Precisely the same skill and experience is essential in identifying and dealing with faulty outputs from AI hallucinations or factual errors.

But that, as they say, is a topic for another day.