Senior Test Manager – AI Assurance, Automation & Quality Engineering - #2188619

Senior Test Manager – AI Assurance, Automation & Quality Engineering

Location: Central Scotland

Working Arrangement: Hybrid, with travel to Glasgow (2 days per week)

Job Type: Permanent or Contract (Inside IR35)

** APPLICANTS MUST BE IMMEDIATELY AVAILABLE TO BE CONSIDERED **

**Due to background-check requirements, applicants must have a minimum of three years' UK address history. **

We are seeking an experienced Senior Test Manager to lead and own quality assurance, test management and assurance activities across complex enterprise technology programmes, with a particular focus on Conversational AI, Generative AI, Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) and agentic AI solutions.

This is a senior leadership role requiring a strong combination of strategic test management, technical understanding, delivery governance and stakeholder engagement.

The successful candidate will be responsible for defining and implementing comprehensive test strategies, managing testing teams, overseeing end-to-end test delivery and ensuring that technology solutions meet agreed quality, performance, security and business acceptance criteria.

A strong understanding of Conversational AI testing (Chat and/or Voice) is essential. Candidates should have experience establishing appropriate testing approaches for AI-enabled solutions, assessing quality risks and ensuring that emerging technologies are adequately tested and assured before production deployment.

The role will involve working closely with AI Engineers, Data Scientists, Software Engineering teams, business stakeholders and senior programme leadership to ensure effective testing governance, transparency of quality outcomes and successful delivery.

Key Responsibilities

1. Test Strategy, Planning & Management

• Define, own and implement the overall test strategy across enterprise technology and AI programmes.
• Establish comprehensive test plans, approaches, schedules, resource models and delivery milestones.
• Manage the full testing lifecycle, including requirements analysis, test preparation, execution, defect management and formal sign-off.
• Define appropriate testing levels, including functional, integration, system, regression, performance, security, UAT and non-functional testing.
• Establish risk-based testing approaches aligned to business priorities, technical complexity and delivery objectives.
• Ensure appropriate requirements traceability, test coverage and evidence-based quality assurance.
• Provide clear visibility of testing progress, delivery risks, dependencies and quality status.
• Ensure testing activities are aligned with programme plans, release schedules and business expectations.

2. AI, Conversational AI & Emerging Technology Assurance

• Lead the test management and assurance approach for Conversational AI solutions across chat and voice platforms.
• Establish testing strategies for LLM-powered applications, chatbots, voice assistants, RAG solutions and agentic AI workflows.
• Ensure appropriate validation of conversational accuracy, contextual understanding, intent recognition, response consistency and multi-turn interactions.
• Oversee testing of voice capabilities, including speech recognition (ASR), text-to-speech (TTS), response latency and interruption handling.
• Establish approaches for evaluating AI-generated responses, including accuracy, relevance, hallucinations, bias and adherence to business requirements.
• Ensure appropriate testing of prompts, retrieval accuracy, context management, integrations and AI guardrails.
• Define AI quality benchmarks, evaluation criteria, regression approaches and measurable quality metrics.
• Work with AI Engineering and Data Science teams to establish appropriate assurance standards and production-readiness criteria.

3. Test Delivery & Team Leadership

• Take accountability for end-to-end test delivery, ensuring testing is completed to agreed timescales, quality standards and budgets.
• Lead, manage and develop test managers, test leads, QA engineers and automation specialists.
• Establish clear roles, responsibilities, delivery expectations and reporting structures across testing teams.
• Manage resource allocation, capacity planning and utilisation across multiple workstreams.
• Provide leadership, mentoring and technical direction to testing teams.
• Manage testing dependencies across business, technology, third-party suppliers and programme delivery teams.
• Identify delivery challenges and proactively manage risks, issues and escalations.
• Drive continuous improvement in testing efficiency, effectiveness and quality outcomes.

4. Test Automation & Engineering Governance

• Define the automation strategy and establish appropriate automation objectives across programmes.
• Oversee the development and maintenance of scalable automation frameworks.
• Ensure effective use of automation technologies, including Selenium, Playwright, Cypress, UFT, Appium and PyTest.
• Drive automation adoption across UI, API, integration, mobile, backend and data validation testing.
• Ensure automated testing is appropriately integrated into CI/CD pipelines using Azure DevOps, Jenkins and GitHub Actions.
• Establish appropriate automation coverage, regression efficiency and continuous testing measures.
• Promote the responsible use of AI-assisted testing tools, including GitHub Copilot and Microsoft Copilot.
• Ensure automation solutions deliver measurable improvements in testing efficiency, coverage and quality.

5. Data, API & Enterprise Testing

• Oversee data quality, integrity, profiling, reconciliation and ETL validation activities.
• Manage testing of APIs, integrations, data migrations and end-to-end enterprise workflows.
• Ensure appropriate validation of AI model performance, data quality, bias, drift and reliability.
• Establish appropriate approaches for database, backend and data platform testing.
• Oversee performance, scalability, resilience and security testing activities.
• Ensure testing considers relevant security principles and established OWASP guidance.
• Manage test evidence, defect resolution and quality acceptance across complex enterprise environments.

6. Governance, Reporting & Stakeholder Management

• Establish and maintain effective testing governance, standards, processes and controls.
• Provide regular testing status reports, dashboards and management information to senior stakeholders.
• Communicate quality risks, defects, testing progress and release recommendations clearly and effectively.
• Define and monitor key quality metrics, including test coverage, defect trends, automation coverage, execution progress and defect leakage.
• Chair testing governance meetings, defect triage sessions and quality review forums.
• Ensure appropriate escalation and resolution of critical defects and delivery risks.
• Provide objective, evidence-based recommendations regarding release readiness and production acceptance.
• Work closely with programme managers, delivery leads, technical architects and business stakeholders.
• Support audit, regulatory and compliance requirements through appropriate testing evidence and governance.