c1.12 Evaluate models and monitor for drift
Assess foundation models against your security and quality requirements before production, and continuously watch for drift or adversarial degradation afterward.
Implementation guidance
How to implement on AWS
Use Amazon Bedrock model evaluation with datasets that include adversarial and jailbreak examples; analyze bias with SageMaker Clarify. Document results and approvals before production, then monitor invocation and guardrail metrics with CloudWatch to catch drift.
Acceptance criteria
A documented model evaluation (including adversarial tests) precedes production; drift/guardrail metrics are monitored with alarms.
Practical implementation
How this control is implemented in each reference architecture:
Model evaluation runs in a non-production account of the OU with adversarial datasets; documented approval gates promotion to the inference account, then CloudWatch watches for drift.
- Run Amazon Bedrock model evaluation in a development account of the Generative AI OU using datasets that include adversarial and jailbreak examples; analyze bias with SageMaker Clarify.Generative AI OU (development account)
Amazon Bedrock 
Amazon SageMaker - Record results and a documented approval as the gate to promote the model/config into the inference (Generative AI) account.Generative AI account Amazon Bedrock
- After promotion, monitor invocation, guardrail-intervention, and quality metrics in CloudWatch and alarm on drift from baseline.Generative AI account
CloudWatch
How to validate: A documented evaluation with adversarial tests precedes production; approval is recorded before promotion; drift/guardrail metrics have alarms.
Avoid: Evaluating a model once and never again. Behavior and threats change; keep the evaluation datasets and re-run on model or config changes, and monitor for drift.
Evaluate before production and monitor for drift within the one account, using a pre-production stage and adversarial test datasets.
- Run Bedrock model evaluation with adversarial/jailbreak datasets against a pre-production configuration; analyze bias with SageMaker Clarify.Generative AI tier (IAM role) Amazon Bedrock Amazon SageMaker
- Document results and require sign-off before flipping the configuration to production use.Application tier (IAM role) Amazon Bedrock
- Monitor invocation, guardrail, and quality metrics in CloudWatch and alarm on drift.In-account logging CloudWatch
How to validate: A documented, adversarial-inclusive evaluation precedes production; drift and guardrail metrics are alarmed.
Avoid: Going to production on vendor benchmarks alone. Test against your own adversarial cases and keep watching for drift after launch.
Threats addressed
- Deploying a model that fails safety, bias, or robustness requirements.
- Undetected model behavior degradation over time.
AWS services
Amazon Bedrock model evaluation Amazon SageMaker Clarify Amazon CloudWatch Framework mappings
OWASP GenAI/LLM Top 10 (2026)
- LLM05 Data and Model Poisoning
- LLM07 Misinformation
NIST AI Risk Management Framework (AI 100-1)
- MEASURE.1 Appropriate methods and metrics are identified and applied
- MEASURE.2 AI systems are evaluated for trustworthy characteristics
AWS Well-Architected Generative AI Lens (Security)
- GENSEC02-BP01 Implement guardrails to mitigate harmful or incorrect model responses
Learning resources
References
- AWS SRA for AI — Capability 1, Model evaluation and validation