Opportunities and Challenges of Explainable AI (XAI) in Health Care: A Review ()
1. Introduction
AI is a very powerful tool for analyzing medical information in the healthcare sector. It helps medical professionals for predicting and recommending appropriate treatments. For improving healthcare services, smart healthcare involves advanced technologies, such as AI, the Internet of Things (IoT), and cloud computing that uses smart devices. Those devices collect healthcare data like, physical activity, heart rate, blood sugar, and sleep habits including others. Therefore, people can monitor their own health record and share with doctors to diagnose diseases more accurately. AI uses these health records to detect diseases at earlier time, screening health and provides suitable treatment plans.
However, most existing AI models based on black box operations cannot clearly explain their decisions or predictions due to a lack of transparency and potential bias, which makes their use risky when human lives are involved [1] [2]. Researchers argue that AI must be explainable before it can be safely used in high-stakes areas like health care [3]. XAI helps AI models become more transparent and understandable by explaining how and why a particular decision was made. This, in turn, helps doctors, healthcare professionals, and researchers trust AI systems more by increasing transparency, predicting results, and improving current models by identifying errors [1] [2]. When it comes to making decisions about individual patients, current researchers believe that these explanations are mostly false and overly optimistic. They argue that these explainability methods are unreliable and limited which can be incomplete, incorrect, or misleading that often do not help doctors in real understanding [3]. Researchers also found abnormalities in ECG or EEG signals and highlights doctor’s notes in clinical tests which require more attention. Recent studies used both Machine Learning (ML) and DL models along with most commonly used XAI methods like: SHapley Additive Explanations (SHAP), Local Interpretable Modelagnostic Explanations (LIME), and Gradient-weighted Class Activation Mapping (GradCAM). Overall, SHAP is the most widely used XAI method for identifying important clinical features when predicting diseases or patient outcomes. It is commonly combined with machine learning models such as XGBoost and Random Forest (RF), which work well with structured clinical data. Grad-CAM explains medical image predictions by highlighting important regions in images using heatmaps generated from DL models like Convolutional Neural Networks (CNNs) [4].
This paper reviews the opportunities and challenges of XAI in healthcare, highlighting current methods, applications, limitations, and future research directions. This review includes coverage period (2020-2025), accuracy, and XAI approaches (SHAP, LIME, Grad-CAM etc.) along with applications.
Our research contributions are as follows:
We present a comprehensive and systematic review of XAI techniques in healthcare applications along with ML and DL models for various clinical data types.
We highlight opportunities and challenges associated with XAI in healthcare applications.
We identify key research gaps for existing XAI methods for clinical text data, and provide some suggessions as future research directions.
The paper is organized as follows: Section 2 describes an overview of the XAI system. Section 3 deals with opportunities of XAI in healthcare system. Section 4 covers recent applications of XAI systems. Section 5 discusses existing challenges of XAI in healthcare. Section 6 highlights recent advancements of XAI in healthcare systems. Section 7 presents the proposed XAI framework for healthcare based on identified research gaps. Finally, Section 8 provides a summary of the research and recommendations for future work.
2. Overview of XAI
Healthcare provides real-time patient monitoring, smart environments with strong privacy protection by using advanced technologies including AI, the Internet of Things (IoT), big data, and others. Some studies exist on XAI in healthcare do not focus enough on how data is analyzed and how model explanations are interpreted, which limits real-world use. The study proposes an XAI-based system architecture for analyzing medical images where federated learning ensures patient’s privacy and XAI model is applied to validate model performance. Their experiment shows that XAI is effective model for real-world healthcare applications [5]. XAI models help doctors and healthcare professionals to trust AI system so that it can be more understandable and predictable [6] [7]. This explanation ensures that AI-based conclusions are meaningful and useful for medical practice.
Explanations can be global or local where Global explanations help to understand the overall behavior of an AI model, while local explanations explain the reason behind a specific prediction. This two combination provides deeper insights into the model’s decisions. Explainability can be classified into two categories: modelagnostic methods, which work with any machine learning algorithm, and model specific methods, which are designed for a particular type of model [8]. The review [9] shows that knowledge graphs improve the comprehensibility of AI systems in healthcare applications by identifying misleading or false information as well as dangerous drug interactions and responses. This knowledge helps bridge the gap between medical experts and AI models. Explanations can be classified as post-hoc or pre-hoc methods, where the post-hoc explanations describe the outcomes generated by an AI system, whereas pre-hoc explanations describe how the AI system functions during the decision-making process [10] [11]. In healthcare, AI systems are evaluated using measures like sensitivity and specificity with higher accuracy requirements than other fields, but in clinical practice, accuracy alone is not sufficient. Doctors need to understand how and why an AI system makes its predictions, as clear explanations help them trust the system and apply its results in real clinical practice. Questions about patient data and features are important for providing clear explanations, although AI models may produce unreliable or unfair predictions for certain training data due to bias [12]. Therefore, the paper [13] identifies that the AI systems should be more explainable and transparent for healthcare professionals.
Another core term interpretability helps developers understand how an AI model makes decisions, while explainability presents these decisions in an understandable way to end-users to build trust, together enabling insight into black-box models and supporting a balance between accuracy and transparency [14]. XAI helps to make an AI application more transparent and also assists in the improvement of the AI application by applying simple cross-domain tools and techniques [15].
For highstake domain like healthcare, transparency is a major issue that allows patients and clinicians to build trust about how and why the system produces a specific output for given inputs. It ensures reliability and accuracy of the AI system and guarantees human-AI collaboration [16].
Fidelity explain how well an explanation affects the true decision-making process. Low fidelity hides the true effect which is dangerous [17].
Common XAI Techniques and Approaches
This section categorizes commonly used XAI methods. The research [18] reviews popular XAI methods such as LIME, SHAP, and DeepLIFT based on organizations about when explanations are given. This helps users choose the right explanation method for different applications. Based on human thinking and cognition, the framework shows how explanations can be designed in a way that helps people build trust in AI systems.
In intrinsic interpretability, AI models are designed in a transparent way so that people can easily understand how decisions are made without needing extra explanation tools [19]. Some examples of intrinsically interpretable AI models include decision trees, which display decisions step-by-step like a flowchart; generalized additive models (GAMs), which make it evident how each input feature affects the outcome; and rule-based systems, which make decisions based on explicit and predetermined rules. These models are easy to understand because their decision process is visible. This intrinsic model ensure high transparency [20]. Post-hoc explainability methods explain AI models after they have already made a decision, especially complex black-box models. For example: LIME, SHAP, and Layer-wise Relevance Propagation (LRP), where LIME explains a single prediction by slightly changing the input data. Then, it observes how the AI’s output changes and creates a simple model that helps people understand why a specific decision was made. Another method, SHAP, explains an AI prediction by assigning a clear importance score to each input feature, showing how strongly each factor influenced the final decision [21]. Deep learning models like LRP explains decisions by tracking the prediction backward through the network layers and highlights the most important features. Although post-hoc methods work well with complex models such as deep neural networks, their explanations can sometimes result in misleading insights [22].
3. Opportunities of XAI in Healthcare
This section highlights how XAI enhances healthcare systems.
3.1. Clinical Decision Support Systems (CDSS)
CDSS are computer-based tools that assist physicians in selecting the best antibiotics for their patients. In addition to helping antimicrobial stewardship programs identify patients who might require treatment modifications, they assist doctors in making safer and more effective antimicrobial treatment decisions [23]. An explainable CDSS also assist doctors in identifying women who are at risk for gestational diabetes mellitus (GDM) [24]. The use of XAI makes the models more transparent in clinical decision making although some real problems arise with implementation and operational speed [25].
3.2. Trust and Adoption of XAI in Healthcare
XAI applications should be transparent and understandable in making decisions for healthcare applications [26]. For obtaining user trust including XAI, requires transparency [27], specially in healthcare sector. Due to limitations of black-box models, XAI systems increases transparency and lowers the possibility of making poor decisions by giving users insight into the decision-making process [7] [28]-[30]. This trust increases the user acceptance in healthcare applications [31].
3.3. Error Detection and Bias Identification of XAI in Healthcare
XAI helps healthcare stakeholders to understand and trust AI so that decisions are non-biased and accurate [32]. XAI explains how decisions are made and the errors and biases can be found and fixed through reviewing and auditing these explanations [33].
3.4. Regulatory Compliance of XAI in Healthcare
Due to “black-box” nature, AI models faces challenges for regulatory compliance. XAI should be appropriately explainable in building trust on AI systems. These explanation ensures regulatory compliance through emphasizing transparency, accountability, and patient safety for taking clinical decisions. These regulations focus on ensuring the ethical and responsible use of AI [34].
3.5. Knowledge Discovery of XAI in Healthcare
XAI enhances transparency and builds trust among stakeholders in the healthcare sector [35]. Knowledge discovery is a crucial process for identifying patient-related risks and other insights that cannot be detected through traditional analytical approaches. XAI supports the monitoring of personal health and the prediction of healthcare conditions [36].
The related opportunities are illustrated in Figure 1.
Figure 1. Opportunities of XAI in healthcare.
4. Recent Applications of XAI in Healthcare
Currently, XAI is extensively used in healthcare applications, including medical imaging, assessment of children’s developmental status, AI safety, electronic health records, medical text processing, personalized treatment planning, risk prediction, disease diagnosis, COVID-19 management, clinical decision support systems, hospital admission prediction, drug response prediction, and pain recognition, as illustrated in Figure 2 [37]-[42].
Figure 2. Recent applications of XAI in healthcare.
5. Challenges of XAI in Healthcare
5.1. Trade-Off between Accuracy and Interpretability
It is very difficult to maintain a better trade-off between explainability and predictive accuracy [43]. The explanations should be designed and customized carefully.
5.2. Healthcare Regulations Framework
In healthcare, transparency is essential as decisions of XAI have a direct impact on people’s lives [44]. Researchers and developers should analyze XAI compliances with healthcare regulations, ethical standards, and regulatory frameworks [45].
5.3. Explanations Reliability
Various types of explanations can influence physicians’ treatment choices. This finding [46] suggests that current ML tools are not sufficiently reliable for recommending improved treatment plans, which may lead to incorrect clinical decisions.
5.4. Lack of Clinical Validation
In healthcare, clinical validation is a primary requirement for the regulatory and certification processes and CDSSs [47]. To achieve clinical validation, prediction accuracy is important. But, AI systems often produce false positive or false negative predictions due to random errors [48].
5.5. Complexity Analysis and Sacalbility Issue
Current XAI models are often very complex, which makes their explanations difficult for users to understand. In addition, generating explanations for large models and datasets requires high computational effort and is usually slow. For instance, Kernel SHAP is a modelagnostic technique that computes explanations by sampling many combinations of input features. This process becomes computationally expensive for high-dimensional data, reducing its scalability for large datasets and complex models [49].
5.6. Human-AI Interaction Issues
The Human-AI Interaction challenge in XAI deals with the explanations produced by AI systems that enable users to interact with and make decisions based on AI outputs.
Figure 3 illustrates the possible challenges of XAI in Healthcare applications.
Figure 3. Possible challenges of XAI in healthcare.
In healthcare, explanation quality should be evaluated based on fidelity, robustness, clinician usefulness, and fairness. The system should be robust and secure for avoiding manipulation of predicting explanations that misleads the healthsector. Also, model’s biasness and excessive trust on explanations could mislead the system [50]-[55]. These criteria ensure that explanations are accurate, consistent, meaningful for clinicians, and free from bias, thereby supporting safe and effective clinical deployment.
6. Recent Developments of XAI in Healthcare
The work [56] estimates volumetric breast density from images without the need for segmentation, using a 3D convolutional neural network (CNN). The model achieved strong agreement with minimal bias. SHAP-based explanations showed that accurate predictions relied on relevant breast tissues and provides reliable breast density estimation along with meaningful explainability. XAI addresses several existing challenging issues like model transparency and ethical concerns, by improving the interpretability and trustworthiness of AI models in healthcare. The study [5] proposes an XAI-driven Healthcare 5.0 architecture and validates its effectiveness through case studies on medical imaging and privacy-preserving electrocardiogram (ECG) monitoring using federated learning technique. The method [57] proposes a healthcare framework that integrates AI, blockchain, and the metaverse for designing an efficient digital healthcare services. Doctors and patients interact via a blockchain-based system, where medical data are securely stored and analyzed using XAI techniques such as GradCAM and LIME. The framework ensures reliable disease detection, data protection, transparency, and interpretability in healthcare systems.
The method [58] uses SHAP and LIME to analyze symptomps and predict severity for COVID-19 data. This study shows their effectiveness in supporting explainability. Also, the study [59] uses centralized and federated learning techniques for classifying heart diseases. Centralized models achieved up to 81.1% accuracy using Naive Bayes (NB), while federated logistic regression (LR) reached 78.2% accuracy, protecting patient privacy. Model predictions are interpretated using SHAP and LIME for finding the potentiality of interpretable heart disease prescreening systems. In order to enhance the heart disease detection system, another study [60] presents a hybrid framework that combines explainable AI, deep learning, and machine learning methods. The framework reduces classification error by 20% - 25% across multiple datasets. It addresses data imbalance, missing values, and feature inconsistencies. This work enhances clinical trust and scalability in healthcare systems. Also, this work delivers high predictive performance for the healthcare systems.
The work [61] integrates Decision Trees (DT), NB, RF, and XGBoost for improving both accuracy and interpretability in predicting the risks of diseases including Diabetes, Anaemia, Thalassemia, Heart Disease, and Thrombocytopenia. The method achieves 99.2% accuracy while model predictions are achieved through SHAP and LIME. The study [37] introduces PersonalCareNet, a new deep learning framework for addressing the lack of interpretability in existing AI healthcare models. It delivers both global and patient-specific explanations by combining CNN with attention mechanisms and SHAP. Also, the study [62] predicts workplace mental health using XAI and ML methods including RF, xGBoost, SVM, and AdaBoost. The xGBoost and RF performed best and achieved high accuracy, while SHAP and LIME provided transparent explanations of some important factors. It includes treatment-seeking behavior and past or present mental health conditions.
This review divides studies into some significant categories such as:
These are used as current research methods that find the most important clinical features for enhancing diagnostic transparency.
Clinical tasks: Disease diagnosis, risk prediction, drug response prediction, and mental health analysis.
Data types: Medical imaging, structured clinical data, ECG/EEG signals, and text data.
Also, existing methods find the risk affecting factors for the patient.
Table 1. Summary of the existing XAI methods in healthcare.
References, year |
Used Methods |
XAI Approaches |
Accuracy |
Healthcare Applications |
H. M. van der Velden et al. [56], 2020 |
CNN |
SHAP |
- |
Breast density estimation |
D. Saraswat et al. [5], 2022 |
CNN, Federated transfer learning |
CAM and Grad-CAM |
98% |
COVID-19 patients |
S. Ali et al. [57], 2023 |
Blockchain |
Grad-CAM, LIME |
- |
Healthcare |
A. Nambiar et al. [58], 2023 |
DT, XGBoost Classifier, and Neural Network Classifier |
SHAP, LIME |
- |
COVID-19 symptom analysis and severity prediction |
Rodriguez and Nafea [59], 2025 |
Linear-kernel SVM model |
SHAP |
83.3% |
Cardiovascular disease detection |
Talukder et al. [60], 2025 |
Multilayer Perceptron (MLP) |
SHAP, LIME |
100.0% |
Heart disease detection |
Agrawal et al. [61], 2025 |
DT, RF, NB |
XGBoost, SHAP, LIME |
99% |
Healthcare |
M. S. Vani et al. [36], 2025 |
CNN + Attention |
SHAP, Grad-CAM, Force Plot, Feature Importance |
97.86% |
Healthcare |
T. Mokheleli et al. [62], 2025 |
RF, XGBoost, SVM, and AdaBoost |
SHAP, LIME |
94% |
Mental health |
Most of the existing XAI approaches likely SHAP, LIME, and GradCAM offer post-hoc explanations. These explanations might not fully reflect the true internal decision-making processes of complex models. Misinterpretation can have serious consequences in healthcare sector which raises the concerns about explanation fidelity. As a result, we propose an XAI framework for healthcare in Section 7 that aims to minimise these existing challenges.
We have discussed existing challenges at Section 5. Based on Table 1, we have summarized some limitations:
Strong clinical validation,
Cross validation,
Bias and overfitting,
And, also, rely on post-hoc explanations.
7. Proposed Framework for XAI in Healthcare
Figure 4 presents the proposed framework for an explainable AI-based healthcare system.
Figure 4. Proposed framework for XAI in healthcare.
This framework addresses the major challenges discussed in Section 5, including the trade-off between accuracy and interpretability, lack of clinical validation, reliability of explanations, scalability issues, regulatory concerns, and human-AI interaction limitations.
The framework starts by collecting the raw healthcare data from clinical databases, medical sensors, electronic health records, and medical imaging devices. Healthcare data are frequently noisy, incomplete, and heterogeneous. Therefore, the preprocessing stage reduces bias and enhances data quality. For this, it performs several operations likely noise reduction, normalization, missing value handling, and data cleaning. This stage directly addresses the challenges of unreliable predictions and model biasness discussed in Section 5.
After preprocessing, feature selection and feature extraction are applied to identify clinically significant attributes. It also reduces dimensionality and computational complexity. This stage improves scalability and reduces the complexity of explanation generation, which helps overcome the scalability and complexity challenges of existing XAI models. Furthermore, selecting clinically meaningful features improves explanation interpretability for healthcare professionals.
The processed features are then used to train machine learning or deep learning models using historical and labeled clinical data. To reduce overfitting and improve generalization performance, cross-validation and performance evaluation are incorporated during model training. This stage addresses the challenge of balancing predictive accuracy and interpretability while improving model robustness and reliability. The optimized features analysis the explanations using classifiers and those explanations guide the generation of model predictions. The explanation evaluation and result validation ensure reliability, fairness, and clinical relevance.
This step helps reduce false-positive and false-negative predictions. It also addresses the lack of clinical validation discussed in Section 5. The validated results generate report that summarizes predictions, explanations, and performance metrics in an interpretable format. Also, this report provides practical advices which supports clinicians and decision-makers in diagnosis, patient management, and treatment planning. The completed results are then delivered to end users.
These steps address the human-AI interaction challenges. Also, it guarantees that explanations remain relevant and trustworthy in real clinical settings. The proposed framework ensures that predictions are not only accurate but also transparent, reliable, and helpful for medical applications by integrating explainability throughout the ML lifecycle.
8. Conclusions and Future Directions
XAI plays a significant role in the healthcare industry. It enhances accountability, transparency, and trust in AI-driven systems. It provides interpretable predictions and clear explanations that enable clinicians and healthcare professionals for validating and integrating the AI system into clinical decision-making. XAI supports informed diagnoses, treatment planning, and patient management. But, there exist several challenges on this area including the trade-offs between interpretability and predictive accuracy, limited clinical relevance of some explanations, data quality and bias issues, and difficulties in integrating XAI tools into existing clinical methods. Also, regulatory, ethical, and validation-related issues further complicate real-world deployment. Future work should focus on hybrid modeling techniques that maintain a better trade-off between explainability and performance, clinician-centered system design, reliable explanations, bias minimization, standard framework, proper clinical validation, cloud-based healthcare, and authorized healthcare regulations for ensuring trustworthy and responsible healthcare system.
Author Contributions
All authors have contributed equally to performing this research.