AI is a very powerful tool for analyzing medical information in the healthcare sector. It helps medical professionals for predicting and recommending appropriate treatments. For improving healthcare services, smart healthcare involves advanced technologies, such as AI, the Internet of Things (IoT), and cloud computing that uses smart devices. Those devices collect healthcare data like, physical activity, heart rate, blood sugar, and sleep habits including others. Therefore, people can monitor their own health record and share with doctors to diagnose diseases more accurately. AI uses these health records to detect diseases at earlier time, screening health and provides suitable treatment plans.

However, most existing AI models based on black box operations cannot clearly explain their decisions or predictions due to a lack of transparency and potential bias, which makes their use risky when human lives are involved [1][2]. Researchers argue that AI must be explainable before it can be safely used in high-stakes areas like health care [3]. XAI helps AI models become more transparent and understandable by explaining how and why a particular decision was made. This, in turn, helps doctors, healthcare professionals, and researchers trust AI systems more by increasing transparency, predicting results, and improving current models by identifying errors [1][2]. When it comes to making decisions about individual patients, current researchers believe that these explanations are mostly false and overly optimistic. They argue that these explainability methods are unreliable and limited which can be incomplete, incorrect, or misleading that often do not help doctors in real understanding [3]. Researchers also found abnormalities in ECG or EEG signals and highlights doctor’s notes in clinical tests which require more attention. Recent studies used both Machine Learning (ML) and DL models along with most commonly used XAI methods like: SHapley Additive Explanations (SHAP), Local Interpretable Modelagnostic Explanations (LIME), and Gradient-weighted Class Activation Mapping (GradCAM). Overall, SHAP is the most widely used XAI method for identifying important clinical features when predicting diseases or patient outcomes. It is commonly combined with machine learning models such as XGBoost and Random Forest (RF), which work well with structured clinical data. Grad-CAM explains medical image predictions by highlighting important regions in images using heatmaps generated from DL models like Convolutional Neural Networks (CNNs) [4].

This paper reviews the opportunities and challenges of XAI in healthcare, highlighting current methods, applications, limitations, and future research directions. This review includes coverage period (2020-2025), accuracy, and XAI approaches (SHAP, LIME, Grad-CAM etc.) along with applications.

Our research contributions are as follows:

We present a comprehensive and systematic review of XAI techniques in healthcare applications along with ML and DL models for various clinical data types.We highlight opportunities and challenges associated with XAI in healthcare applications.We identify key research gaps for existing XAI methods for clinical text data, and provide some suggessions as future research directions.

The paper is organized as follows: Section 2 describes an overview of the XAI system. Section 3 deals with opportunities of XAI in healthcare system. Section 4 covers recent applications of XAI systems. Section 5 discusses existing challenges of XAI in healthcare. Section 6 highlights recent advancements of XAI in healthcare systems. Section 7 presents the proposed XAI framework for healthcare based on identified research gaps. Finally, Section 8 provides a summary of the research and recommendations for future work.

Healthcare provides real-time patient monitoring, smart environments with strong privacy protection by using advanced technologies including AI, the Internet of Things (IoT), big data, and others. Some studies exist on XAI in healthcare do not focus enough on how data is analyzed and how model explanations are interpreted, which limits real-world use. The study proposes an XAI-based system architecture for analyzing medical images where federated learning ensures patient’s privacy and XAI model is applied to validate model performance. Their experiment shows that XAI is effective model for real-world healthcare applications [5]. XAI models help doctors and healthcare professionals to trust AI system so that it can be more understandable and predictable [6][7]. This explanation ensures that AI-based conclusions are meaningful and useful for medical practice.

Explanations can be global or local where Global explanations help to understand the overall behavior of an AI model, while local explanations explain the reason behind a specific prediction. This two combination provides deeper insights into the model’s decisions. Explainability can be classified into two categories: modelagnostic methods, which work with any machine learning algorithm, and model specific methods, which are designed for a particular type of model [8]. The review [9] shows that knowledge graphs improve the comprehensibility of AI systems in healthcare applications by identifying misleading or false information as well as dangerous drug interactions and responses. This knowledge helps bridge the gap between medical experts and AI models. Explanations can be classified as post-hoc or pre-hoc methods, where the post-hoc explanations describe the outcomes generated by an AI system, whereas pre-hoc explanations describe how the AI system functions during the decision-making process [10][11]. In healthcare, AI systems are evaluated using measures like sensitivity and specificity with higher accuracy requirements than other fields, but in clinical practice, accuracy alone is not sufficient. Doctors need to understand how and why an AI system makes its predictions, as clear explanations help them trust the system and apply its results in real clinical practice. Questions about patient data and features are important for providing clear explanations, although AI models may produce unreliable or unfair predictions for certain training data due to bias [12]. Therefore, the paper [13] identifies that the AI systems should be more explainable and transparent for healthcare professionals.

Another core term interpretability helps developers understand how an AI model makes decisions, while explainability presents these decisions in an understandable way to end-users to build trust, together enabling insight into black-box models and supporting a balance between accuracy and transparency [14]. XAI helps to make an AI application more transparent and also assists in the improvement of the AI application by applying simple cross-domain tools and techniques [15].

For highstake domain like healthcare, transparency is a major issue that allows patients and clinicians to build trust about how and why the system produces a specific output for given inputs. It ensures reliability and accuracy of the AI system and guarantees human-AI collaboration [16].

Fidelity explain how well an explanation affects the true decision-making process. Low fidelity hides the true effect which is dangerous [17].

This section highlights how XAI enhances healthcare systems.

Currently, XAI is extensively used in healthcare applications, including medical imaging, assessment of children’s developmental status, AI safety, electronic health records, medical text processing, personalized treatment planning, risk prediction, disease diagnosis, COVID-19 management, clinical decision support systems, hospital admission prediction, drug response prediction, and pain recognition, as illustrated in Figure 2 [37]-[42].

Figure 2.Recent applications of XAI in healthcare.

The work [56] estimates volumetric breast density from images without the need for segmentation, using a 3D convolutional neural network (CNN). The model achieved strong agreement with minimal bias. SHAP-based explanations showed that accurate predictions relied on relevant breast tissues and provides reliable breast density estimation along with meaningful explainability. XAI addresses several existing challenging issues like model transparency and ethical concerns, by improving the interpretability and trustworthiness of AI models in healthcare. The study [5] proposes an XAI-driven Healthcare 5.0 architecture and validates its effectiveness through case studies on medical imaging and privacy-preserving electrocardiogram (ECG) monitoring using federated learning technique. The method [57] proposes a healthcare framework that integrates AI, blockchain, and the metaverse for designing an efficient digital healthcare services. Doctors and patients interact via a blockchain-based system, where medical data are securely stored and analyzed using XAI techniques such as GradCAM and LIME. The framework ensures reliable disease detection, data protection, transparency, and interpretability in healthcare systems.

The method [58] uses SHAP and LIME to analyze symptomps and predict severity for COVID-19 data. This study shows their effectiveness in supporting explainability. Also, the study [59] uses centralized and federated learning techniques for classifying heart diseases. Centralized models achieved up to 81.1% accuracy using Naive Bayes (NB), while federated logistic regression (LR) reached 78.2% accuracy, protecting patient privacy. Model predictions are interpretated using SHAP and LIME for finding the potentiality of interpretable heart disease prescreening systems. In order to enhance the heart disease detection system, another study [60] presents a hybrid framework that combines explainable AI, deep learning, and machine learning methods. The framework reduces classification error by 20% - 25% across multiple datasets. It addresses data imbalance, missing values, and feature inconsistencies. This work enhances clinical trust and scalability in healthcare systems. Also, this work delivers high predictive performance for the healthcare systems.

The work [61] integrates Decision Trees (DT), NB, RF, and XGBoost for improving both accuracy and interpretability in predicting the risks of diseases including Diabetes, Anaemia, Thalassemia, Heart Disease, and Thrombocytopenia. The method achieves 99.2% accuracy while model predictions are achieved through SHAP and LIME. The study [37] introduces PersonalCareNet, a new deep learning framework for addressing the lack of interpretability in existing AI healthcare models. It delivers both global and patient-specific explanations by combining CNN with attention mechanisms and SHAP. Also, the study [62] predicts workplace mental health using XAI and ML methods including RF, xGBoost, SVM, and AdaBoost. The xGBoost and RF performed best and achieved high accuracy, while SHAP and LIME provided transparent explanations of some important factors. It includes treatment-seeking behavior and past or present mental health conditions.

This review divides studies into some significant categories such as:

Used Models: CNN, Random Forest, and XGBoost. Used XAI: SHAP and LIME.

These are used as current research methods that find the most important clinical features for enhancing diagnostic transparency.

Clinical tasks: Disease diagnosis, risk prediction, drug response prediction, and mental health analysis.Data types: Medical imaging, structured clinical data, ECG/EEG signals, and text data. Also, existing methods find the risk affecting factors for the patient.

Table 1.Summary of the existing XAI methods in healthcare.

Most of the existing XAI approaches likely SHAP, LIME, and GradCAM offer post-hoc explanations. These explanations might not fully reflect the true internal decision-making processes of complex models. Misinterpretation can have serious consequences in healthcare sector which raises the concerns about explanation fidelity. As a result, we propose an XAI framework for healthcare in Section 7 that aims to minimise these existing challenges.

We have discussed existing challenges at Section 5. Based on Table 1, we have summarized some limitations:

Strong clinical validation,Cross validation,Bias and overfitting, And, also, rely on post-hoc explanations.

Figure 4 presents the proposed framework for an explainable AI-based healthcare system.

Figure 4.Proposed framework for XAI in healthcare.

This framework addresses the major challenges discussed in Section 5, including the trade-off between accuracy and interpretability, lack of clinical validation, reliability of explanations, scalability issues, regulatory concerns, and human-AI interaction limitations.

The framework starts by collecting the raw healthcare data from clinical databases, medical sensors, electronic health records, and medical imaging devices. Healthcare data are frequently noisy, incomplete, and heterogeneous. Therefore, the preprocessing stage reduces bias and enhances data quality. For this, it performs several operations likely noise reduction, normalization, missing value handling, and data cleaning. This stage directly addresses the challenges of unreliable predictions and model biasness discussed in Section 5.

After preprocessing, feature selection and feature extraction are applied to identify clinically significant attributes. It also reduces dimensionality and computational complexity. This stage improves scalability and reduces the complexity of explanation generation, which helps overcome the scalability and complexity challenges of existing XAI models. Furthermore, selecting clinically meaningful features improves explanation interpretability for healthcare professionals.

The processed features are then used to train machine learning or deep learning models using historical and labeled clinical data. To reduce overfitting and improve generalization performance, cross-validation and performance evaluation are incorporated during model training. This stage addresses the challenge of balancing predictive accuracy and interpretability while improving model robustness and reliability. The optimized features analysis the explanations using classifiers and those explanations guide the generation of model predictions. The explanation evaluation and result validation ensure reliability, fairness, and clinical relevance.

This step helps reduce false-positive and false-negative predictions. It also addresses the lack of clinical validation discussed in Section 5. The validated results generate report that summarizes predictions, explanations, and performance metrics in an interpretable format. Also, this report provides practical advices which supports clinicians and decision-makers in diagnosis, patient management, and treatment planning. The completed results are then delivered to end users.

These steps address the human-AI interaction challenges. Also, it guarantees that explanations remain relevant and trustworthy in real clinical settings. The proposed framework ensures that predictions are not only accurate but also transparent, reliable, and helpful for medical applications by integrating explainability throughout the ML lifecycle.

XAI plays a significant role in the healthcare industry. It enhances accountability, transparency, and trust in AI-driven systems. It provides interpretable predictions and clear explanations that enable clinicians and healthcare professionals for validating and integrating the AI system into clinical decision-making. XAI supports informed diagnoses, treatment planning, and patient management. But, there exist several challenges on this area including the trade-offs between interpretability and predictive accuracy, limited clinical relevance of some explanations, data quality and bias issues, and difficulties in integrating XAI tools into existing clinical methods. Also, regulatory, ethical, and validation-related issues further complicate real-world deployment. Future work should focus on hybrid modeling techniques that maintain a better trade-off between explainability and performance, clinician-centered system design, reliable explanations, bias minimization, standard framework, proper clinical validation, cloud-based healthcare, and authorized healthcare regulations for ensuring trustworthy and responsible healthcare system.

All authors have contributed equally to performing this research.