TITLE:
Beyond Black-Box Benchmarking: A Statistically Validated, Interpretable, and Uncertainty-Aware Machine Learning Framework for Insurance Premium Estimation
AUTHORS:
Jude Isememe Itoyah
KEYWORDS:
Insurance Premium Prediction, Machine Learning, Interaction Feature Engineering, SHAP, Explainability, Quantile Regression, Stacked Ensemble, Bayesian Optimization
JOURNAL NAME:
Journal of Computer and Communications,
Vol.14 No.7,
July
22,
2026
ABSTRACT: The application of machine learning to insurance premium estimation has accelerated considerably in recent years. However, the field faces three structural weaknesses. These include over-reliance on black-box models without explainability, absence of credible statistical validation, and failure to quantify actuarial uncertainty around point predictions. This research addresses all three limitations within a unified, end-to-end experimental framework using a dataset collected from Kaggle. Eight interaction and polynomial features grounded in actuarial domain knowledge are engineered from the retrieved dataset. An out-of-fold (OOF) stacked ensemble is proposed, combining Linear Regression, Random Forest, XGBoost, and LightGBM as base learners under a Ridge meta-learner. All models are evaluated using a 10-fold cross-validation protocol with Wilcoxon signed-rank statistical testing. There are critical findings from the ML experimentation. With sufficiently rich interaction features, a simple Linear Regression model achieves R2 = 0.8827 on the held-out test set, statistically indistinguishable from the stacked ensemble (R2 = 0.8825). All baseline models were trained on the original monetary expense scale, while a log1(1 + y) transformation was applied solely to support interaction feature construction. SHAP values are therefore expressed directly in monetary units (£) without requiring back-transformation. These results challenge the prevailing assumption in the literature that insurance premium prediction fundamentally requires non-linear architectures. Also, SHAP analysis reveals that five of the six most influential features are interaction terms, with smoker × BMI contributing a mean absolute SHAP value of approximately £3215 per prediction, which is 50% more than the marginal smoking effect alone. Quantile regression produces prediction intervals covering 69.8% of test instances, with widths increasing disproportionately in high-risk strata. The framework is benchmarked against six existing works and offers improvements in predictive validity, interpretability, and statistical rigour on this benchmark dataset.