How do you find great testers for great testing?

Explore Assurity 11 August 2014 Adam Howard

Never having been involved in recruiting testers before, Sean was suddenly entrusted with finding five testers in a year to build on Trade Me’s existing team of seven. Given this organically assembled team were the acknowledged ‘connectors of the business’ through their knowledge and passion, figuring out how to add to a winning formula was a crucial challenge.

Launching into the process of advertising their roles, Sean and his team found themselves swamped with applications, with around 40 CV’s hitting their desks for each advertised position. On the surface, this might sound like a good thing – plenty of choice! – but not so, says Sean.

He found himself just flicking through the CV’s ‘panning for gold’ – the little nuggets that suggested a candidate might have something to offer. Everything else was simply discarded, but this didn’t sit too well with Sean – to casually discard applicants’ carefully constructed CV’s just didn’t seem fair.

He also found that the volume of applicants was not only draining, but also meant that they were rushing the process, which was impacting on the results. Sean would find himself interviewing an applicant three weeks after reading their CV and so there was no real sense of continuity that enabled a clear and considered selection process.

Finally, they recognised that when they were interviewing the candidates, they were asking themselves the wrong question. Instead of thinking “Are they good enough?”, they realised they needed to be asking “Are they GREAT?”

And this meant that they had to take a step back and actually define what ‘great’ meant to Trade Me. And in spite of Sean’s initial gut instinct – “I’m great, so I’ll hire people like me!” – this was a task that they took very seriously and has become the cornerstone of Trade Me’s successful recruiting process.

In the same way that testers apply quality criteria to a software product, Trade Me identified quality criteria that they could apply to their applicants. This meant identifying what the qualities of a great tester were for them, but they also considered why those qualities were important.

After some consideration, they had identified some key qualities that they felt great testers should have such as:

  • Communication: a tester should be able to describe, explain and justify their approach and their thinking
  • Openness: testers need to be able to share their ideas and concerns to build effective relationships with a range of people
  • Loyalty: Trade Me has a strong culture and team spirit, so they want someone who can buy into that and become a loyal and committed team member
  • Passionate: really great testers are those who are passionate about what they do, so they wanted people with a passion for testing and a passion for Trade Me
  • Courageous: we all have successes and failures and they recognised that good testers are those who aren’t afraid to fail, instead learning from their mistakes

Having developed a view of what ‘great’ meant, Sean and his team were ready to deploy a new approach to trying to find people who fitted that bill. They implemented a process that was designed to help them achieve exactly that.

The first stage still involved screening those CV’s. But instead of randomly panning for vague nuggets of gold, they now had a good idea of what their great candidates should look like on paper. Defining this more explicitly also meant that they were able to enlist the HR team and recruiters to assist at this stage in recognising those attributes – rather than requiring Sean or other testers to drive by instinct in picking out the promising CV’s.

They also started thinking critically about how they pulled in these CV’s in the first place. They tested their job listings, asking their existing staff – i.e. people like those they wanted to hire – to evaluate whether they would apply for that job. This helped them recognise that job ads that were too short would attract a vast swathe of applicants who felt they could satisfy the minimal stated criteria, whereas those that were too long scared often detail-oriented tester-types away.

Equally, they found that cover letters could be quite divisive. They tried asking applicants to answer specific questions – “What is exploratory testing?” or “What are the benefits of test automation?” – but found that all too often people simply ignored these questions. Frustratingly, applicants seemed unwilling to put in the same time and effort to secure a job that Trade Me was putting in to attract them!

As well as a typical ‘panel interview’ with questions that focus on a candidate’s experiences and achievements around projects and teams, those applicants who pass the screening process earn a one-on-one chat with Sean, who again focuses on the identified characteristics of greatness. Over a coffee in a café, Sean stressed the importance of this face-to-face time in getting a true feeling for the personality of a candidate.

He described how the choice to run these interviews in a café creates an interesting social dynamic – people are far more willing to engage and speak openly in a casual environment, but this does contrast with most people’s expectations of a traditional, formal, even interrogative interview experience.

Similarly, Sean likes to challenge the applicants’ expectations of their role, suggesting he would often describe Trade Me’s fast-paced delivery model where the testers are the release managers as a sort of trick. If someone applying for a testing role wasn’t either slightly nervous or ambitiously excited by that challenge, he would be concerned!

Those candidates not sent running by Sean’s interrogation next face a technical test of their practical skills and abilities, another part of the hiring process that evolved over time. Initially, candidates were simply tested and had to meet a certain number of ‘correct’ answers to make the grade. But they soon realised that this wasn’t a very constructive method of understanding the person’s skills and thinking.

So the technical test was revamped to better resemble Trade Me’s actual test process where all testing is peer reviewed. The candidate would be given a challenge and would then have the opportunity to explain and justify their approach, which was far more useful in exposing their thinking and their underlying mindset.

Tests like this are also useful for identifying those instances where a slightly over-eager candidate might have over-emphasised their skills on a CV. Sean recounted a memorable experience where an applicant was staring hopelessly at an automation tool in which they had claimed proficiency when one of 2013’s large earthquakes struck Wellington giving Sean an excuse to mercifully usher them from the building!

Such disasters (personnel or natural!) aside, the final stage is to give candidates an opportunity to meet and greet with existing members of Trade Me’s team. Meeting fellow testers as well as developers, BA’s and anyone else with whom they will interact in their role, gives both sides a chance to see if there is a good team fit. Can they actually work with each other?!

There are plenty of obstacles at this stage too, as Sean described one candidate who had eagerly researched the various backgrounds of those people he was meeting and proceeded to engage them in conversation about random details from their past. He had all the skills to be an excellent asset to the organisation, but, after the meet and greet, the rest of the team were a bit too scared by his knowledge of their lives to want him to join them!

With this final hurdle cleared, candidates are either welcomed into the fold or given clear feedback as to why they were not successful, which was something Sean was clearly passionate about – that those people who aren’t quite right should get some sort of justification and explanation that can help them hone their applications in future.

Ultimately, the clear message that shone through in Sean’s ER was that hiring great testers is a long and involved process that can’t be approached lightly. Sean’s insights and answers to questions in the Open Season discussion demonstrated that he – and others at Trade Me – have really spent a lot of time studying how they attract their testers and this dedication has seemingly paid off for them as their team has rapidly and successfully expanded in the last two years.

The evening concluded with a lively session of Open Season discussion, with the 30+ attendees eager to learn from Sean’s experience and each other – the room comprised of those looking for tips and tricks to deploy on both sides of the interview table.

Among the many ‘horror stories’ – an interview experience where the panel asked zero questions, a hire who ‘ran away’ after it transpired they had neither working knowledge of a computer nor a valid work visa, and a selection of terribly clichéd ‘stock’ interview questions (that are nonetheless useful on occasion) – there were also many useful tips and discussions.

There was debate around the benefit of asking philosophical questions in interviews and how you can use this technique to dig down into the mindset of a tester – important because it cannot be so easily taught as practical skills. There was also a discussion of the risks and rewards of hiring based on potential – “treat potential as a project”, advised Sean – to ensure that it is realised, rather than abandoned and ignored.

There was also some healthy back and forth on the value recruiters can bring to the process of hiring testers, with the diversity offered by their alternate sample selections weighed against their limited ability to grasp and identify the qualities that are truly important to an organisation. The group did suggest that this could be combatted by inviting recruiters to sit in on a few interviews to gauge your priorities.

The relative merits and redundancy of reference checks also came up, with some suggesting that we only ever list ‘positive’ references. A few people countered that often references we think will be positive might actually go against us! Naturally, the identified solution was to ‘test’ your reference by having a friend ring them posing as a potential employer to validate the slant of the reference they provide!

Overall, it was another very enjoyable workshop, with many first time attendees getting involved and a valuable mix of experience and humour enlightening everyone with some ideas and skills relating to the trials and tribulations of hiring great testers.

Thanks to Assurity for sponsoring the evening and providing a venue and refreshments. Register now for the next WeTest Workshop, a hands-on Testing Challenge on Wednesday 10 September. Be quick as there are just a few spots left.

If you have suggestions for topics that you would like to see covered in future workshops, send these through to Adam Howard.