ISTQB® Certified Tester AI Testing

This course provides comprehensive training for the ISTQB Certified Tester AI Testing certification. It covers fundamental concepts of AI, significantly focussing on Machine Learning and also, quality characteristics specific to AI-based systems, and various testing techniques and methods applicable to AI-based systems.

The course contents will include detailed explanations, practical exercises, and hands-on activities for each topic covered in the course outline.

By the end of this course, participants will be able to:

• Understand the fundamental concepts of AI and machine learning.
• Identify and explain quality characteristics specific to AI-based systems.
• Apply various testing techniques and methods to AI-based systems.
• Evaluate and improve the performance of AI models.
• Address challenges in testing AI-based systems, including bias, transparency, and explainability.

1     Introduction to AI

• Definition of AI and AI Effect
• Narrow, General, and Super AI
• AI-Based and Conventional Systems
• AI Technologies
• AI Development Frameworks
• Hardware for AI-Based Systems
• AI as a Service (AIaaS)
• Pre-Trained Models
• Standards, Regulations, and AI

2   Quality Characteristics for AI-Based Systems

• Flexibility and Adaptability
• Autonomy
• Evolution
• Bias
• Ethics
• Side Effects and Reward Hacking
• Transparency, Interpretability, and Explainability
• Safety and AI

3     Machine Learning (ML) Overview

• Forms of ML
• ML Workflow
• Selecting a Form of ML
• Factors Involved in ML Algorithm Selection
• Overfitting and Underfitting

4     ML Data

• Data Preparation as Part of the ML Workflow
• Training, Validation, and Test Datasets in the ML Workflow
• Dataset Quality Issues
• Data Quality and its Effect on the ML Model
• Data Labelling for Supervised Learning

5 ML Functional Performance Metrics

• Confusion Matrix
• Additional ML Functional Performance Metrics for Classification, Regression, and Clustering
• Limitations of ML Functional Performance Metrics
• Selecting ML Functional Performance Metrics
• Benchmark Suites for ML

6 ML Neural Networks and Testing

• Neural Networks
• Coverage Measures for Neural Networks

7     Testing AI-Based Systems Overview

• Specification of AI-Based Systems
• Test Levels for AI-Based Systems
• Test Data for Testing AI-Based Systems
• Testing for Automation Bias in AI-Based Systems
• Documenting an AI Component
• Testing for Concept Drift
• Selecting a Test Approach for an ML System

8     Testing AI-Specific Quality Characteristics

• Challenges Testing Self-Learning Systems
• Testing Autonomous AI-Based Systems
• Testing for Algorithmic, Sample, and Inappropriate Bias
• Challenges Testing Probabilistic and Non-Deterministic AI-Based Systems
• Challenges Testing Complex AI-Based Systems
• Testing the Transparency, Interpretability, and Explainability of AI-Based Systems
• Test Oracles for AI-Based Systems
• Test Objectives and Acceptance Criteria

9     Methods and Techniques for the Testing of AI-Based Systems

• Adversarial Attacks and Data Poisoning
• Pairwise Testing
• Back-to-Back Testing
• A/B Testing
• Metamorphic Testing
• Experience-Based Testing of AI-Based Systems
• Selecting Test Techniques for AI-Based Systems

10   Test Environments for AI-Based Systems

• Test Environments for AI-Based Systems
• Virtual Test Environments for Testing AI-Based Systems

11   Using AI for Testing

• AI Technologies for Testing
• Using AI to Analyse Reported Defects
• Using AI for Test Case Generation
• Using AI for the Optimization of Regression Test Suites
• Using AI for Defect Prediction
• Using AI for Testing User Interfaces

Attendees intending to take the ISTQB Certified Tester AI Testing examination must hold the ISTQB Certified Tester Foundation Level certificate.

To effectively participate in the technical exercises, delegates should set themselves up in advance of the course, on Google Colab or a similar platform as this will enhance the learning experience and provide the necessary tools to complete hands-on exercises effectively.

After completing this course attendees may consider:

• ARTiBA Certified AI Engineering for Business and Management
• ISTQB Certified Tester Advanced Level Test Analyst
• ISTQB Certified Tester Advanced Level Technical Test Analyst

This course is designed for:

• Testers, test analysts, data analysts, test engineers, test consultants, test managers, user acceptance testers, and software developers involved in testing AI-based systems and/or AI for testing.
• Project managers, quality managers, software development managers, business analysts, operations team members, IT directors, and management consultants seeking a basic understanding of testing AI-based systems and/or AI for testing.