NZQA passes exams with flying colours every year

SITUATION

“What most people don’t know is that running secondary exams from the start to the day we publish the results is an 18-month cycle which touches so many people that we can say we have an impact on the lives of just about every New Zealander,” says Sinziana Ciubotaru, Senior Project Manager, Information Services, NZQA.

The biggest day of the year for students and NZQA is results publication, when months’ worth of internet traffic avalanches the organisation’s website in minutes. This is the point when NZQA knows if it has done the job right… or not – and there is a lot at stake. “It is a massive undertaking and from the moment the exams are created to the results being delivered, all systems must be in place and work as expected. There can be no negative impact, despite the fact that in each cycle, hundreds of changes and updates are made,” continues Sinziana.

Prior to Assurity's engagement and the provision of its comprehensive testing services, website and other system issues could result in people being unable to connect view their results immediately. In today’s climate, when social media has dramatically increased scrutiny on the one hand and the load on NZQA systems on the other owing to amplification, the stakes are higher.

SOLUTION

Assurity, Sinziana says, is instrumental in NZQA’s success and has been for 10 years. “Together we do quite a bit of testing so we can be sure that our systems work as expected before they go into production.”

In practice, Assurity provides performance testing across all NZQA systems with a particular focus on results publication day. This is a significant event for the organisation (and the country) in which the equivalent of months’ worth of logins and user activity ‘hit’ in the minutes and hours when exam results go live.

It may sound simple – just test the maximum number of concurrent users, right? – but nothing could be further from the truth, Sinziana points out. “Probably the biggest challenge is complexity and change. Our systems aren’t constant and every exam cycle is different. The number of systems we have interacting is so substantial that getting someone up to speed on all the dependencies and intricacies can take months or even years. And then, at the same time, there is a major push towards digitisation and digital transformation. These changes are really like changing the engines of a jet in flight.”

This makes people who grasp NZQA’s data flows, exceptions and broader partner ecosystem valuable, she continues.

A further area in which Assurity provides thought and delivery leadership is Targeted Exploratory Testing. This addresses the volume of work necessary for continual quality assurance and is considered a key component of the NZQA delivery model. “Exploratory testing has accelerated the process of evaluating complex intertwined systems with innovative approaches to testing” says Sinziana.

Explaining that Assurity has demonstrated an ability to think outside the box and think big picture, she says examples include planned ‘Just in Time’ testing where team members log in as users and use systems just ahead of go live, with a consistent ability to identify and resolve issues before they have real-world impacts. “With the nature of exams, this is crucial – we simply can’t only test well in advance because come exam day, those tests just aren’t relevant any longer.”

Assurity has also created a dashboarding system which replaces previous disparate reports and spreadsheets tracking performance. By creating a consolidated visual system, NZQA personnel enjoy immediate insight on how exams, results and other essential metrics are tracking. The system is so successful that it is today published on public screens within the company – attesting to the confidence NZQA enjoys that its systems can be relied upon.

RESULTS

The sheer duration of the engagement NZQA has maintained with Assurity is a demonstration of the consistent and dependable value delivered. Sinziana says the testing services are the Authority’s guarantee that it delivers a quality product which protects the integrity of the qualifications it administers. “Stability is essential for our organisation and that’s what Assurity provides. This isn’t an easy ask as we have convoluted systems with a challenging number of touch points and dependencies.”

As the organisation migrates to the cloud along with a major middleware upgrade, Sinziana says testing has proven crucial not only to business as usual, but also maintaining operations with zero impact. “It’s gone smoothly and that is a testament to the comprehensive approach testing. Nothing is left to chance and that means we have no surprises – not on results day and not when making substantial changes to our systems and infrastructure.”

Praising the work ethic and diligence built into the testing services, she says the Assurity team is comprised of "curious people who contribute to our journey, constantly challenging and looking to add value". NZQA CIO Shane Gaskin notes Assurity’s mature approach to customer engagement.

Finally, Gaskin says “At a time where NZQA is rethinking its testing and automation approach and looking for more added value in other areas of continuous improvement, Assurity are working closely with us to ensure we are focused on business outcomes beyond just testing”.

We’re building internal testing capability including the introduction of an automated performance testing environment. The fact that Assurity is willing to share their knowledge and help us create a strong team of testers says a lot about their view on the creation and exchange of lasting value”

Shane Gaskin, CIO, NZQA