TITLE:
Applications of Large Deviation Theory to Massive Data Models: A Numerical Approach for Measuring the Rarity of Extreme Events
AUTHORS:
Bou Diop
KEYWORDS:
Large Deviation Theory, Rare-Event Probability Estimation, Adaptive Importance Sampling, Convex Optimization, Rate Function Estimation
JOURNAL NAME:
Journal of Applied Mathematics and Physics,
Vol.14 No.7,
July
31,
2026
ABSTRACT: The exponential growth of data in modern systems increases the need for robust statistical methods capable of describing rare but critical events. This paper proposes a computational methodology based on large deviation theory to efficiently estimate rare-event probabilities in large-scale systems. We introduce a coherent approach that integrates a solid theoretical framework with advanced numerical algorithms for estimating rate functions. Unlike standard Monte Carlo methods, our approach leverages convex optimization and adaptive importance sampling, which significantly reduces variance and accelerates estimator convergence, as demonstrated by our numerical results. These results attest to the numerical robustness and practical utility of our method in diverse fields such as quantitative finance, telecommunications networks, and machine learning.