Accelerating software testing with AI in high compliance environments

In my role as a Test Lead, I see firsthand how constant changes to enterprise systems impact delivery teams. Whether driven by new budgets, regulatory updates or internal enhancements, large enterprise organisations operate in a state of continuous delivery.

This creates a continuous cycle of requirements. Historically, our analysis and scenario creation processes relied heavily on the time of dedicated business system testers and subject matter experts, who had to manually create and format accurate test scenarios.

While working recently with a large enterprise client, my team faced a critical technical challenge: how could we leverage artificial intelligence to accelerate the development of high-quality test scenarios while strictly maintaining the data security and compliance protocols required by a highly regulated organisation?

The security and architecture constraints

Deploying large language models within a high compliance environment presents significant security and architectural hurdles. We selected Assurity Intelligence because it provides a sandboxed, secure environment specifically designed for trialling AI capabilities safely.

The architectural points of difference were critical for our client:

• Data isolation: The selected language models do not have direct access to the internet. They rely purely on previously trained data alongside the specific information the user provides in the prompt. This ensures all data remains completely offline, giving the enterprise absolute protection from external security risks.
• Prompt governance: Our engineering teams retain the ability to trial and review all preset prompt templates. This gives us overarching control over the application and standardises how the models process complex system requirements.

Methodology: Session-based testing

To evaluate the models rigorously, we utilised structured and time-boxed session-based testing. We sat down with enterprise domain specialists who applied requirements from their respective areas of expertise to generate and review scenarios in real time.

During the pilot, we intentionally kept any form of knowledge repository off-limits to the models. We needed to establish a baseline of how the models performed using only the in-prompt information, without uploading detailed internal business documentation.

We evaluated two distinct prompt templates: one focused on generating high-level test scenarios and another attempting to build detailed test cases.

Technical findings and the context window

The feedback from the domain specialists gave us valuable technical insights into AI test generation.

We discovered that the high-level scenario prompt produced significantly higher-quality outputs than the detailed test case prompt. The scenario outputs were highly usable and contained far fewer hallucinations than the detailed test case prompts. Because the models did not have access to an internal knowledge repository, they lacked the deep business context required to accurately generate highly granular test cases.

A key insight from our pilot was identifying the context window sweet spot. We had to find the optimal balance between the volume of requirements and the detail processed by the models without compromising output accuracy. The domain specialists found that supplying smaller, focused chunks of requirements yielded significantly more robust results than processing extensive, detailed documentation.

Ultimately, the domain specialists reported that using these optimised AI prompts saved them a significant amount of evaluation and generation time, providing an excellent foundational starting point for test creation.

Next steps in AI maturity

Assurity Intelligence has now been deployed to the initial group of business users within a core delivery domain at the client organisation. Representatives from these teams have even established a Change Champion group to provide ongoing operational support.

Our next technical phase focuses on expanding these capabilities through internal knowledge repositories. By securely contextualising the specific business domain, we believe AI models can bridge the gap between high-level scenarios and detailed test execution, further reducing quality assurance timelines while maintaining absolute data integrity.

Successfully accelerating quality assurance with AI requires robust data governance, structured experimentation and continuous domain expert validation. When grounded in these principles, I have seen AI transform software testing from a bottleneck into a strategic enabler, delivering high-confidence software quality at speed across high-compliance environments.

If you are keen to learn more about how Assurity Intelligence can add value to your organisation, please contact our team for a conversation.