Model drift: why AI systems decay, and how to catch it early

Traditional software does the same thing tomorrow that it did today. AI systems do not. Their behaviour depends on the data they see, and the data an organisation produces changes constantly: new products, new suppliers, new document formats, new customer behaviour. Over time, a model validated on last quarter's reality becomes a little less right about this quarter's.

What drift looks like in practice

• A document reader that was reliable on the old waybill layout struggles with a new supplier's format
• A demand forecast trained before a pricing change keeps predicting the old pattern
• A screening model sees a new kind of role and scores candidates less consistently
• An assistant built on one vendor's model behaves differently after the vendor updates it

None of these produce an error message. The system keeps running and keeps producing answers. The answers simply get worse, slowly enough that nobody notices until trust has already gone.

Drift is silent. That is exactly why it has to be measured.

How UniversAIMS catches it

Every system we deploy has a validated baseline from launch. UniversAIMS compares live behaviour against that baseline, task by task, and tracks the gap over time. Small movements are expected. Sustained or sudden movement raises an alert to our engineers, with the affected inputs attached.

1. BaselineAccuracy on representative tasks is recorded and signed off at launch.
2. CompareLive outputs are sampled and compared against the baseline continuously.
3. AlertSustained or sudden movement is flagged to an engineer with examples.
4. CorrectThe system is retrained, recalibrated or re-routed, then re-validated.

Fixing drift without breaking trust

Correcting a drifting system is not just a technical step. Staff who noticed odd results need to know they were heard and that the system is right again. That is why drift corrections appear in the monthly review with leadership, alongside the ROI figures they affect.