ISTQB® Certified Tester AI Testing

The ISTQB® CT‑AI course provides a concise introduction to AI, ML, and the methods used to test AI‑based systems. Participants learn to assess data quality, test ML models, evaluate GenAI and LLMs, and apply AI‑specific testing aligned with ISO/IEC 25059.

It covers the full AI/ML lifecycle, with hands‑on exercises in model creation, performance evaluation, data testing, metamorphic testing, red teaming, and LLM exploratory testing.

The course contents will include detailed explanations, practical exercises, and hands-on activities for each topic covered in the course outline.

By the end of this course, participants will:

• Understand fundamental concepts, capabilities, and limitations of Generative AI in testing.
• Develop practical skills in prompt engineering for software test tasks.
• Identify and mitigate risks of using GenAI, including hallucinations, reasoning errors, and data privacy issues.
• Explore LLM-powered test infrastructure and operational practices (LLMOps, fine-tuning).

Chapter 1 – Introduction to AI (120 minutes)

• AI‑based vs conventional systems
• Narrow, general, and super AI
• Types of AI technologies
• Generative AI
• Hardware for ML systems
• Development & hosting of AI models
• ML development frameworks
• Regulations & standards for AI

Chapter 2 – Quality Characteristics for AI‑Based Systems (45 minutes)

• ISO/IEC 25059 AI quality characteristics
• AI functional correctness
• AI robustness
• Transparency, user controllability, intervenability
• Societal & ethical risk mitigation
• Acceptance criteria for AI‑based systems

Chapter 3 – Machine Learning (375 minutes)

• ML forms: supervised, unsupervised, reinforcement learning
• ML workflow: data prep - model training - tuning - testing - deployment - monitoring
• Data acquisition, preprocessing, feature engineering
• Training/validation/test datasets; cross‑validation
• ML functional performance metrics (accuracy, precision, recall, F1)
• Neural networks and coverage measures
• Hands‑on exercises (H1/H2):
• Build an ML model
• Perform data preparation
• Evaluate metrics
• Explore effects of different models/dataset combinations

Chapter 4 – Testing AI‑Based Systems (195 minutes)

• Locked vs adaptive AI systems
• Statistical approaches to testing
• The test oracle problem
• Testing GenAI systems
• Red teaming (K3 level)
• Exploratory testing of LLMs (H2)
• Test levels for ML systems
• Risk‑based testing

Chapter 5 – Input Data Testing (180 minutes)

• Input data risks and mitigations
• Testing for bias
• Data pipeline testing
• Data representativeness testing
• Dataset constraint testing
• Label correctness testing
• Hands‑on exercise: input data testing (H2)

Chapter 6 – Model Testing for ML Systems (225 minutes)

• ML model risks
• Reviewing ML model documentation
• ML functional performance testing (probabilistic systems)
• Adversarial testing
• Metamorphic testing (H2)
• Drift testing
• Overfitting & underfitting
• A/B testing
• Back‑to‑back testing

Chapter 7 – ML Development Testing (30 minutes)

• ML development risks
• API testing
• Deployment testing: installability, rollback, canary, shadow, cross‑device
• Model conversion testing

• ISTQB® Foundation Level (CTFL) certification (mandatory)
• Recommended: 6+ months in software testing, software development, or data‑related roles

This course is ideal for professionals involved in the testing or development of AI‑based systems, including:

• Testers, test analysts, test engineers, test consultants
• Test managers and QA professionals
• Data analysts and data scientists
• Software developers working on AI
• User acceptance testers
• Technical managers, product owners, and project managers
• Quality managers and IT leaders seeking an understanding of AI‑testing challenges

It is also appropriate for anyone seeking foundational knowledge of how AI‑based and ML‑based systems are tested.

• ISTQB® Foundation Level
• ISTQB® Advanced Level – Test Analyst
• ISTQB® Advanced Level – Technical Test Analyst
• ISTQB® Specialist Level – Test Automation Engineer
• ISTQB® Specialist Level – Testing with Generative AI