A good model is not enough: you also need to know when it stops being good.
The initial goal was a classic one: predict the commercial premium of a car insurance policy. The most interesting part came afterwards, when the models were confronted with more recent data: data drift degrades performance far faster than one imagines.
Approach
- Exploratory analysis to identify the factors that weigh on premiums: vehicle age, bonus-malus coefficient, driver age.
- Two families of models: GLMs (Tweedie, Gamma) for interpretability, LightGBM for raw performance, compared on RMSE, MAE, R² and MAPE.
- Reproducible pipeline: cleaning, training, evaluation, then drift analysis on later data.
What the drift revealed
On the recent data, 45% of the key variables had changed distribution. Direct consequence: the RMSE rises by 147% and the R² drops from 0.75 to -0.35, in other words the model becomes worse than a simple average. Changes in driver profiles (age, bonus-malus) weigh far more than the static characteristics of the vehicles.
Recommendations
- Continuous monitoring of input distributions and production metrics.
- Periodic retraining triggered by drift thresholds rather than on a fixed date.
- Interpretability and performance are not opposites: keep a GLM as a readable reference next to the boosting model.
Stack
Python, Pandas, NumPy, Scikit-learn, LightGBM, Statsmodels, Matplotlib and Seaborn.