Every change gets the eval.
AI vendors update models, change defaults, add features and retire old versions, often with little notice. Treat each change like a software release.
What changes
Without Asking You
- Model versions and behaviour
- Default settings
- Prices and usage limits
- Features added or removed
- Older versions retired
Your routine
Every Time
- Watch release notes for tools you use
- Rerun your eval set (Day 87)
- Roll out to one team first
- Fix the version where the tool allows it
- Know how to roll back
Test every change before customers do.