Early fraud detection systems relied heavily on predefined rules and statistical thresholds. These systems, often encoded in if-then logic, flagged transactions based on absolute deviations from norms, e.g., unusually large amounts, foreign transactions, or time-of-day anomalies. While these systems offered high interpretability and low latency, they were brittle and inflexible. Their primary limitations include:

The advent of machine learning introduced models such as Decision Trees, Random Forests, Logistic Regression, and Support Vector Machines (SVMs). These methods showed improved accuracy and generalization. Notably, ensemble methods like Gradient Boosted Decision Trees (GBDT), including XGBoost and LightGBM, became popular due to their robustness and interpretability. However, these models still struggled to:

Deep learning brought the ability to learn complex, non-linear relationships from raw and structured data. Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks captured sequential dependencies in transaction histories. Convolutional Neural Networks (CNNs) were adapted to learn from feature maps of user behavior. Autoencoders provided unsupervised learning techniques for anomaly detection. While powerful, deep learning models suffered from:

Fraud schemes often involve multiple entities operating in collusion. Graph-based methods, including Graph Neural Networks (GNNs), became popular for modeling relationships among users, merchants, and devices. Link prediction and community detection algorithms are used to detect fraud rings. However, challenges persist in terms of scalability and the need for labeled graph data.

Unsupervised anomaly detection techniques, such as Isolation Forests, One-Class SVMs, and Autoencoders, are widely used when labeled fraud data is sparse. These models detect deviations from normal behavior. However, their effectiveness is often hindered by:

To overcome the limitations of individual paradigms, hybrid models have emerged that combine multiple detection techniques. For instance, a supervised model may provide primary classification while an unsupervised model flags edge cases for review. Ensemble methods further improve performance through aggregation strategies.

The growing regulatory demand for transparent decision-making has spurred interest in Explainable AI (XAI). Techniques such as SHAP, LIME, and counterfactual reasoning have been used to understand feature contributions and justify model predictions. Despite progress, integrating XAI into real-time fraud systems remains a work in progress.

Our Contribution Relative to Existing Work: Our proposed system integrates the strengths of all the above approaches by:

The proposed architecture consists of five key layers:

This layered structure allows the system to handle vast volumes of heterogeneous financial transactions while offering adaptability to institutional constraints and regional compliance mandates.

Figure 1 illustrates the complete pipeline from data ingestion through risk-aware feature engineering, hybrid AI inference, interpretability, and action integration with fraud operations platforms.

1) Data Ingestion and Normalization Layer

This layer supports real-time and batch ingestion from multiple transaction sources:

Data pipelines are built using Apache Kafka, Amazon Kinesis, and RESTful APIs. Ingested data undergoes deduplication, normalization, encryption, and tagging based on institution, channel, and geography.

2) Risk Feature Engineering Module

This is a critical component where domain expertise is translated into engineered features. It processes raw transaction streams into a feature matrix using:

These features are fed into a Risk Scoring Function (RSF) that computes a probabilistic fraud risk score for each transaction.

3) Hybrid AI Inference Layer

This dual-model stack consists of:

The outputs are fused using an ensemble decision function that dynamically adjusts thresholds based on risk tier, time-of-day, and user segment. Our use of an unsupervised component is grounded in comparative evaluations of anomaly detection algorithms for multivariate data [4] .

4) Interpretability and Compliance Layer

Integrated XAI components provide per-transaction explanations using SHAP values. A counterfactual reasoning module evaluates how feature adjustments would change outcomes, aiding in regulatory audits. We provide per-transaction and global explanations using SHAP [3] and, where appropriate for analyst workflows, complementary model-agnostic explanations via LIME [5] . These controls align with modern guidance on interpretable ML and finance-grade anomaly explanations [9] [10] .

5) Decisioning and Integration Layer

Final fraud probability scores are sent to downstream systems:

Responses are timestamped, logged, and pushed to a feedback loop for continuous learning.

The pipeline ingests data from a wide variety of sources, including:

We use Apache Kafka and Amazon Kinesis for real-time data streaming. Each Kafka topic is partitioned by transaction type, institution ID, region, and channel (e.g., web, mobile, ATM), enabling parallel processing and routing efficiency.

Transformation is handled by a multi-stage ETL (Extract, Transform, Load) pipeline built using:

Key enrichment processes include:

Processed features are stored in:

This bifurcation enables efficient model training (offline) and inference (online) workflows.

The model stack is deployed and managed using:

Models are retrained periodically or upon drift detection (see below), with rollback capabilities using canary deployment patterns.

Robust observability is built into the pipeline using:

Key metrics monitored include:

With this infrastructure in place, the system is capable of scaling to hundreds of millions of transactions per day, while maintaining low-latency fraud scoring, auditability, and model governance.

Rather than relying solely on raw transaction fields (e.g., amount, timestamp), we create derived and engineered features that embed expert heuristics and multi-dimensional correlations. Our goal is to enable the model to reason about:

We engineered >250 production-grade features across six families. Below, we list representative features and their construction to improve reproducibility.

A. Behavioral (Per-Entity, Rolling Windows)

$\sum \Delta t\le 24hamount)/24$

$p95inter-txdelta\left(60d\right)/currentinter-txdelta$

B. Contextual (Event and Environment)

C. Identity/Device

D. Temporal

E. External Risk Signals

F. Derived Aggregates/Peer Signals

Normalization & Leakage Control: Continuous features are standardized by robust scalers (median/IQR) computed on training windows only. Categorical targets (e.g., mcc_risk_score) use nested time-series CV to avoid peeking. All feature definitions, SQL/UDFs, and window sizes are versioned in a central registry.

Risk Scoring Function (RSF): The pre-model RSF is

RSF (x) = ∑i wi fi (x)

with wi initialized from SHAP global importances and re-estimated quarterly via constrained least squares subject to monotonicity on selected features (e.g., ip_reputation, mcc_risk_score). RSF serves as a fallback and a decision aid.

The RSF aggregates key risk signals into a normalized risk score (0 - 1) per transaction, which is then passed as a primary input to both the supervised and unsupervised models. It is defined as:

RSF(txn) = Σ (wᵢ · fᵢ(txn)), for i = 1 to n

where:

The RSF provides:

RSF thresholds are calibrated dynamically based on segment (e.g., VIP customers), time-of-day, and geolocation.

To maintain feature reliability across environments and data distributions:

Features that include personally identifiable information (PII) are masked, tokenized, or replaced with pseudonyms prior to storage. All feature transformations are explainable and reproducible, complying with GDPR’s “right to explanation” and financial data minimization principles.

In summary, our feature engineering strategy bridges domain knowledge with statistical rigor, enabling AI models to detect nuanced fraud behaviors while supporting model transparency, drift resilience, and regulatory compliance.

Our model stack is composed of two primary layers:

Primary Detection Layer (Supervised):

Utilizes XGBoost, a gradient-boosted decision tree algorithm known for its performance and explainability. This layer is trained on historical labeled transaction data, including fraud and non-fraud examples. The model outputs a calibrated probability score for each transaction, representing its fraud likelihood based on learned patterns.

Secondary Anomaly Detection Layer (Unsupervised):

Deploys an Autoencoder neural network, trained to reconstruct normal transaction patterns using only non-fraud data. Anomalies are detected when reconstruction error exceeds a defined threshold, indicating that the transaction deviates significantly from normal behavior—even if it wasn’t part of a known fraud pattern. The behavior-modeling role of the autoencoder aligns with deep-learning approaches studied for fraud behavior analysis [6] .

The outputs of the two layers are fused using a weighted ensemble strategy. The final fraud probability PfraudP_{fraud} Pfraud is calculated as:

P_fraud = α · P_xgb + β · (1 - RE_AE)

where:

Thresholds for flagging transactions are dynamically adjusted based on:

For each prediction, especially from the supervised layer, we compute SHAP values to quantify the contribution of each feature. These are visualized for investigators through a dashboard showing:

The unsupervised model’s reconstruction map is also presented visually to highlight which attributes most deviated from the norm.

To prevent adversarial manipulation (e.g., model evasion via crafted transactions), the architecture includes:

Model updates follow a three-step protocol:

Models are monitored for performance decay, and a feedback loop enables retraining based on analyst review outcomes and false positive feedback.

This hybrid AI design delivers the best of both paradigms: supervised accuracy and unsupervised generalization—while supporting traceability, stability, and adaptability in fast-evolving fraud landscapes.

Dataset Composition and Diversity: The evaluation uses anonymized transactions from two multinational banks operating in distinct regulatory regions with different consumer behaviors and payment mixes. The corpus spans card-present (POS/ATM) and card-not-present (web/mobile/e-commerce) channels and covers a broad merchant spectrum (hundreds of MCCs), cross-border remittances, and both urban and rural catchments. This heterogeneity captures 1) region-specific velocity patterns and time-of-day habits, 2) channel artifacts (e.g., device fingerprints and IP hygiene for mobile/web vs. terminal telemetry for POS/ATM), and 3) merchant-category risk gradients that vary by geography.

Representativeness Checks: Prior to modeling, we verified that 1) per-channel transaction share, 2) per-MCC distribution tails, and 3) user-level session statistics (median inter-transaction deltas, weekday/weekend ratios) are materially different across the two institutions—justifying the claim that the dataset encodes regional and channel diversity rather than a single-market snapshot.

External Validity Plan: To further strengthen generalization:

Preprocessing (Unchanged Except Clarifications): We retain time-based splits (70/15/15 train/val/test), SMOTE + undersampling for class imbalance, PII tokenization, and target encoding for high-cardinality keys. All steps are deterministic and versioned.

XGBoot

Tuned with Bayesian optimization (via Optuna), exploring hyperparameters such as tree depth, learning rate, column sampling ratio, and minimum child weight. Evaluation metrics included F1-score, AUC-ROC, and precision at top k.

Autoencoder

Architected with an input layer matching the feature space, symmetric encoder-decoder structure, ReLU activations, and dropout regularization. The reconstruction loss used was Mean Squared Error (MSE). The anomaly threshold was derived from validation-set reconstruction error percentiles.

Regularization Techniques Included:

To ensure robustness, we used time-series aware cross-validation with rolling windows across historical time slices. This strategy ensured the model learned general temporal behaviors and adapted to fraud evolution.

...and so on.

Each fold produced SHAP interpretation metrics to evaluate feature stability over time.

We report the standard classification metrics (Precision, Recall, F1, AUC-ROC, PR-AUC), latency under load, and Top-K alerting effectiveness. In addition, we include a quantitative fairness audit.

Fairness metric. We compute Equal Opportunity Difference (EOD), the absolute difference in true positive rate (TPR) between protected groups g and a reference group r:

EOD = ∣TPRg − TPRr∣

We evaluate EOD across 1) region groups (the two institutions/regions) and 2) account types (consumer vs. small-business), using stratified bootstrap (1000 resamples) to derive 95% CIs.

Results. On the held-out test windows, the Hybrid model achieved:

Both are below the pre-set guardrail of 0.05. Where local drifts transiently pushed EOD near the guardrail, post-training threshold recalibration by segment (Platt/temperature scaling) nudged TPR parity back within bounds without materially changing overall F1. We also monitor predicted-positive rates by group to ensure operational workload parity.

Post-training, each model underwent an explainability audit:

These artifacts are archived and versioned to support external audits and internal governance reviews.

Models were selected not just based on performance but on criteria such as:

Only models meeting all criteria were certified for production rollout.

We benchmark three systems: 1) Legacy Rule-Based, 2) XGBoost, and 3) Hybrid (XGBoost + AE). The table of headline metrics (Precision/Recall/F1/AUC/PR-AUC/Latency/Top-1% hits) remains as reported; below, we make the legacy baseline fully explicit for independent assessment.

Legacy Rule-Based Engine (Rules, Thresholds, Coverage):

Firing logic: A transaction is blocked if ≥2 rules fire; it is queued for review if exactly one fires. Deduplication of multi-rule hits is applied before metric computation.

Observed baseline performance. Using this specification, the baseline achieved a Recall of 0.63 and Precision of 0.78 on the same test windows, matching the summary table and providing a concrete comparator for the learning systems. The learned models subsume and generalize these heuristics while reducing noise from brittle thresholds.

We tested model inference under a simulated transaction load of 10,000 transactions per second, representative of peak fintech environments.

Latency metrics (Hybrid Model):

Description: Figure 2 shows A violin plot showing latency distributions for XGBoost, Autoencoder, and Hybrid model configurations:

To simulate fraud operations, we evaluated top-K alerting effectiveness:

This helps reduce analyst workload by focusing attention on the most suspicious cases.

We monitored feature and prediction stability for 90 days post-deployment. Mean PSI across top features remained <0.1; prediction confidence stayed within ±5% of validation baselines; the autoencoder maintained >85% reconstruction accuracy. Retraining was scheduled at day ~60 when a minor covariate shift emerged in channel-mix features.

Beyond 90 days: handling concept drift and adversarial evasion. After the 90-day horizon, we transition to a sliding-window governance: 1) rolling re-fit of calibration layers every 14 days; 2) drift sentinels combining PSI/JSD with ADWIN change-point tests; 3) seasonality-aware benchmarks that compare like-for-like weeks (e.g., holiday periods). To counter evasion tactics, we continuously red-team the stack with three attack families—slow-roll escalation (gradually increasing amounts), burst laundering (bot swarms of low-value transactions), and device farming (frequent device/IP churn). Signals from these tests feed an adversarial replay suite and trigger canary retrains with cost-sensitive losses plus mild adversarial augmentations. Thresholds and the RSF are re-optimized jointly with business constraints (alert budgets, hold/block SLAs), ensuring robustness without degrading analyst workload.

These experimental results validate our architecture’s ability to balance speed, accuracy, and interpretability in real-time financial fraud detection settings.

The supervised XGBoost model integrates SHAP (Shapley Additive Explanations) to compute feature-level attributions per transaction. Each prediction is accompanied by a breakdown of which features contributed most to the probability of fraud.

Key outputs include:

These explanations are consumable not only by data scientists but also by fraud investigators and auditors.

We developed a lightweight UI module called RiskView, accessible via web browser or embedded into fraud ops tools. It provides:

For selected transactions, we generate counterfactual examples to assist in human interpretation and regulatory review. These scenarios show:

This functionality enhances trust and provides a path for disputability.

We enforce model governance and audit-readiness via:

Monthly reports are generated summarizing:

Baseline traceability: We persist rule-firing events (rule IDs, thresholds, and timestamps) alongside model scores to allow auditor-visible decomposition of why the legacy engine or the AI flagged a transaction.

Fairness traceability: Monthly fairness reports include EOD by segment with bootstrap CIs, trend charts, and any applied post-hoc calibration diffs. Reports are archived with model lineage (MLflow run IDs) and data snapshots to support reproducible re-audits.

Our explainability infrastructure aligns with key regulatory expectations:

We have pre-audited explanation logs and simulation playbooks available for internal and third-party reviewers.

Through this layered transparency infrastructure, our system balances cutting-edge AI with institutional accountability, enabling both operational speed and ethical responsibility.

The model stack is exposed through a RESTful and gRPC API layer, integrated with transaction processing systems. It supports:

Each inference call returns:

Average response time under production load: 128 ms (p95).

All components are stateless, horizontally scalable, and regionally distributed.

In case of model drift or outage:

Data snapshots are archived in S3 and mirrored in staging environments to enable rapid rollback or hot-patching.

With this deployment design, the system meets SLAs for latency, uptime, and auditability in both cloud-native and hybrid enterprise settings.