c2.9 Monitor customized models for drift and bias
Continuously watch customized models in production for drift, bias, and performance degradation that could indicate compromise or quality loss.
Implementation guidance
How to implement on AWS
Use Amazon SageMaker Model Monitor and SageMaker Clarify to track data/quality drift and bias on customized models, with CloudWatch metrics and alarms on deviation from baselines. Re-evaluate when significant drift is detected.
Acceptance criteria
Model Monitor/Clarify run on customized models; baselines and alarms exist; drift triggers re-evaluation.
Practical implementation
How this control is implemented in each reference architecture:
SageMaker Model Monitor and Clarify watch customized models in the Generative AI account for drift and bias, alarming on deviation and triggering re-evaluation.
- Enable SageMaker Model Monitor and Clarify on the customized model in the Generative AI account to track data/quality drift and bias.Generative AI account
Amazon SageMaker - Set baselines and CloudWatch alarms on deviation, and route alarms to the response workflow.Generative AI account
CloudWatch - On significant drift, trigger re-evaluation (link back to the model-evaluation gate) before continuing to serve.Generative AI account Amazon SageMaker
Amazon Bedrock
How to validate: Model Monitor/Clarify run on the customized model; baselines and alarms exist; a drift event triggers re-evaluation.
Avoid: Deploying a customized model and never watching it again. Drift and emerging bias can signal poisoning or quality loss; monitor and re-evaluate.
Model Monitor and Clarify run in-account on the customized model, with baselines, alarms, and a re-evaluation trigger on drift.
- Turn on SageMaker Model Monitor and Clarify for the customized model to track drift and bias.Generative AI tier (IAM role) Amazon SageMaker
- Define baselines and CloudWatch alarms on deviation.In-account logging CloudWatch
- Re-evaluate the model when significant drift is detected.Generative AI tier (IAM role) Amazon SageMaker Amazon Bedrock
How to validate: Monitoring runs on the customized model; baselines/alarms exist; drift triggers re-evaluation.
Avoid: Treating a customized model as static. Watch for drift and bias continuously and re-evaluate when they appear.
Threats addressed
- Undetected model drift or bias emerging after deployment.
- Degradation that could signal poisoning or compromise.
AWS services
Amazon SageMaker Model Monitor Amazon SageMaker Clarify Amazon CloudWatch Framework mappings
OWASP GenAI/LLM Top 10 (2026)
- LLM05 Data and Model Poisoning
- LLM07 Misinformation
NIST AI Risk Management Framework (AI 100-1)
- MEASURE.2 AI systems are evaluated for trustworthy characteristics
- MEASURE.3 Mechanisms for tracking AI risks are in place
- MANAGE.4 Risk treatments, monitoring and response are documented and applied
AWS Security Hub — AI Security Best Practices
- SageMaker.13 SageMaker model quality job definitions should have inter-container traffic encryption enabled
- SageMaker.15 SageMaker model bias job definitions should have inter-container traffic encryption enabled
- SageMaker.25 SageMaker model quality job definitions should have network isolation enabled
AWS Well-Architected Generative AI Lens (Security)
- GENSEC03-BP01 Implement control plane and data access monitoring
Learning resources
References
- AWS SRA for AI — Capability 1, Model evaluation and validation