Auckland Transport’s Open Loop Testing: Approach and lessons learned

Auckland Transport’s (AT) launch of the Open Loop project was the first step toward realising the National Ticketing System (NTS). AT’s mission was simple, yet incredibly complex: to allow commuters to tap their contactless payment cards, or mobile wallets, while keeping the existing HOP card system running flawlessly.

As the Test Manager on this landmark initiative, I’m thrilled to share our journey. This project required a blend of traditional rigour and agile adaptability. We needed to ensure every single tap worked perfectly, whether it was a physical card, phone, or smart watch, and that the financial reconciliation behind the scenes was flawless. Here’s a look at the methodology, the phases we tackled, and the vital insights we gathered.

Our test approach: Rigour meets velocity

Development and testing was structured around the V-model methodology, which provided comprehensive coverage, ensuring every requirement was traceable from initial design to final deployment. Although the delivery model was ultimately a “big bang” launch, where the entire, integrated software suite went live at once, we leveraged opportunities for smaller, iterative testing on specific components. This included hardware upgrades, PCI PAL integration, and the new Open Loop web applications.

Our strategy was risk-driven. The test team focused on areas with the highest potential impact on system quality and the greatest consequence for our business and customers. If a tap failed or if reporting was incorrect, the impact would be immediate and severe.

To maintain control, AT implemented clear quality gates after every key test phase. No phase progressed without a formal sign-off from the project sponsor, based on detailed test plans, results, and sound recommendations. Daily stand-ups and robust defect triage sessions with our multiple vendors were critical for maintaining momentum and resolving issues efficiently.

Focused testing phases

A major organisational focus was reusability. Since Open Loop is just the first of three planned projects moving towards NTS, we consciously designed our tests and documentation for easy adaptation in subsequent phases.

Our testing was officially divided into three major stages:

1. User Acceptance Testing (UAT): This foundational phase ensured the system met real-world needs. We didn’t limit this to the project team; we involved key players from our three customer support teams and a broad base of end-users. Their feedback was essential for refining the commuter experience.
2. Production Verification Testing (PVT): This was the deep dive into system integrity, divided into three critical sub-stages. First, a controlled phase involving major card providers (Visa, Mastercard, AMEX, UnionPay) to validate compliance. Second, a limited-scope beta test phase with a select group of users and NZ banking representatives. Third, a final phase concentrating purely on finance, reporting, and reconciliation.
3. Production Readiness Testing (PRT): The final stamp of quality, ensuring the production environment was robust and scaled correctly ahead of the official launch.

We supported these efforts with both manual and automated testing. Automation was specifically targeted for regression testing, maintaining the stability of the existing HOP card system while we layered the new Open Loop functionality on top. Finally, system security was handled entirely by a third party, guaranteeing an independent and expert review of our architecture and payment handling processes.

Valuable lessons learned and key recommendations

Post project, our retrospective yielded several actionable insights, especially regarding our environment strategy and project management tooling.

Environment management and access

The limitations of our test environments proved to be a significant hurdle. Our key takeaway is that a clear, upfront understanding of these environments is critical.

• Dedicated Architect: We strongly recommend engaging a dedicated architect or a similar technical resource early to conduct a thorough analysis of all test environments – identifying what works well, what has restrictions, and what dependencies exist.
• Access Strategy: An access strategy needs to be established and validated much earlier. Ensuring local and remote access is ready for all test phases is not a trivial task and shouldn’t be left until the last minute.

Streamlining tooling and resources

We relied on Azure DevOps for test and defect management, which proved effective. However, we identified areas to streamline our processes:

• Wider Tool Access: Expanding access to our business Subject Matter Experts (SMEs) for test execution and to the wider project team for reviewing deliverables is crucial for collaboration and transparency.
• Centralised Beta Issues: We realised a single, dedicated inbox or channel for all beta testing issues would have dramatically streamlined our defect management, reducing errors caused by missed or duplicated feedback.
• Dedicated Tool Resource: We recognised the need to hire a dedicated resource to manage the test tool effectively (e.g., Azure DevOps). This would free up the Technical Project Manager to focus on sprint-related technical tasks, optimising their time.

Project planning and communication

Even with a strong team, planning and communication gaps can slow things down.

• Common Project Plan: Moving forward, we recommend utilising a common, accessible project plan for all leads. This single source of truth ensures everyone is working from the same, up-to-date document, enhancing collaboration and reducing confusion.
• Requirements in Azure DevOps: Managing requirements, user stories, and acceptance criteria within sprints in Azure DevOps from the very start would have better facilitated traceability and improved the overall project flow for the testing team.

By proactively adopting these changes, we can streamline future projects, increase efficiency, and deliver even higher-quality software to our customers. This Open Loop experience has provided invaluable insights that will undoubtedly shape our approach as we continue our journey toward a more advanced, open, and convenient transport network for Auckland.