Deep Learning for Predicting Post-Stroke Cognitive Decline Using Multimodal Data: A Synthetic Proof-of-Concept Study ()
1. Introduction
Stroke is a leading cause of disability worldwide and remains a major contributor to long-term cognitive and functional impairment [1]-[4]. Cognitive impairment following stroke is one of the most disabling sequelae, with estimates suggesting that 40% - 60% of survivors develop some degree of cognitive decline within the first year [5]-[8]. This significantly limits independence, hampers rehabilitation, and increases both caregiver burden and healthcare costs. Early identification of patients at high risk of post-stroke cognitive decline (PSCD) is therefore critical for optimizing rehabilitation strategies and improving long-term outcomes [9]-[10]. However, current prediction methods rely primarily on clinical scales such as the National Institutes of Health Stroke Scale (NIHSS) and the Montreal Cognitive Assessment (MoCA), which, while useful, provide limited accuracy when applied in isolation [11]-[13].
Neuroimaging and neurophysiological biomarkers present an opportunity to move beyond these limitations. Structural magnetic resonance imaging (MRI) provides detailed insights into lesion burden and white matter integrity, while electroencephalography (EEG) captures functional disruptions through oscillatory activity and network-level dynamics [14]-[17]. Clinical risk factors such as age, comorbidities, and acute stroke severity add complementary predictive value. Yet, integrating these heterogeneous modalities into a unified framework remains technically challenging, limiting the translation of multimodal biomarkers into clinical practice. Deep learning has emerged as a powerful approach for predictive modelling in neurology, enabling automatic feature extraction from high-dimensional and complex datasets. Although significant advances have been made in stroke lesion segmentation and outcome prediction, most prior models remain unimodal, focusing either on neuroimaging or tabular clinical features [18]-[21]. The potential of multimodal deep learning where imaging, electrophysiological, and clinical data are jointly modelled remains largely unexplored for PSCD prediction. In this proof-of-concept study, we propose a multimodal deep learning pipeline that integrates MRI, EEG, and clinical data for PSCD risk stratification using synthetic data. By combining heterogeneous modalities within a unified architecture, we aim to demonstrate the feasibility and value of integrative modelling for advancing precision neurology in stroke rehabilitation.
2. Methods
2.1. Synthetic Dataset
To evaluate the feasibility of multimodal deep learning for post-stroke cognitive decline (PSCD) prediction in the absence of large annotated clinical datasets, we constructed a synthetic cohort of 800 patients. The dataset was designed to mimic realistic distributions of imaging, electrophysiological, and clinical risk factors observed in post-stroke populations, while maintaining controlled generative mechanisms that allow reproducibility and transparent validation. The high prevalence of cognitive decline (~0.91) was intentionally chosen to simulate a clinically realistic high-risk stroke rehabilitation setting, such as elderly cohorts with severe ischemic burden or tertiary neurorehabilitation centres. This design emphasizes sensitivity and risk stratification rather than population screening.
MRI volumes (32 × 32 × 32 voxels): Each patient was assigned a three-dimensional brain volume. Background signal was simulated with Gaussian noise to represent non-lesioned brain tissue. Lesion-like abnormalities were generated as Gaussian blobs of varying size, intensity, and location, reflecting heterogeneity in infarct burden across patients [22]-[24]. Critically, lesion burden was correlated with outcome probability: patients assigned to the cognitive decline class were more likely to present with larger, higher-intensity lesions, whereas preserved patients showed fewer or smaller lesion-like patterns. This design approximates clinical evidence linking lesion load and white matter integrity with cognitive impairment.
EEG time-series (8 channels × 256 samples): Electrophysiological recordings were simulated to reflect characteristic frequency shifts associated with cognitive decline. Signals were generated as linear combinations of oscillatory components corresponding to canonical EEG rhythms:
Delta (0.5 - 4 Hz) and theta (4 - 8 Hz) enhanced in impaired patients, representing cortical slowing often observed in post-stroke cognitive dysfunction [25]-[27].
Alpha (8 - 12 Hz) and beta (13 - 30 Hz) attenuated in impaired patients, consistent with loss of normal rhythmic activity [28]-[30]. Channel-specific variability and Gaussian noise were introduced to approximate inter-individual and electrode-level variation. The final representation comprised eight parallel channels, each with 256 temporal samples.
Clinical features (12 variables): A tabular set of 12 patient-level features was simulated to capture well-established demographic and vascular risk factors:
Age (continuous, older patients more likely impaired).
Stroke severity (NIHSS), with higher scores associated with worse prognosis.
Education years, representing cognitive reserve, protective in the preserved group.
Comorbidities: hypertension (HTN), diabetes mellitus (DM), and atrial fibrillation (AF).
Lifestyle factor: smoking.
Baseline cognition (MoCA), strongly predictive of post-stroke decline.
Stroke characteristics: laterality (left, right, bilateral, one-hot encoded) and time since stroke_(days).
Distributions were biased such that patients labelled as impaired were more likely to present with advanced age, higher NIHSS, comorbidities, and lower education and MoCA scores, reflecting real-world epidemiological patterns.
Outcome labels: Binary outcome labels (cognitive decline vs preserved cognition) were generated via a logistic risk function that combined MRI lesion burden, EEG spectral composition (delta/alpha ratio), and clinical features. Random noise was added to avoid deterministic separability, introducing variability analogous to unmeasured confounding in clinical data [31]-[33]. The overall positive case rate was ~0.91, which intentionally simulated a high-prevalence clinical setting in which PSCD is common.
Outcome labels were generated using a probabilistic logistic risk model to avoid deterministic separability. Specifically, the probability of post-stroke cognitive decline for subject
was defined as
where
denotes normalized MRI lesion burden,
the EEG delta–alpha power ratio,
baseline MoCA score,
age, and
NIHSS score. Coefficients were selected to reflect clinically plausible effect directions reported in prior PSCD literature, while Gaussian noise
was added to model unobserved confounders and measurement variability. Final labels were sampled from a Bernoulli distribution parameterized by this probability.
Data partitioning: The dataset was split into 70% training, 15% validation, and 15% testing subsets. Stratification ensured similar class distributions across sets. This design enabled iterative model optimization on the training set, unbiased hyperparameter tuning on validation, and final performance estimation on an unseen test set.
2.2. Model Architecture
To integrate heterogeneous data streams (structural imaging, electrophysiological signals, and clinical features), we developed a three-branch multimodal deep learning architecture with late fusion. Each branch was tailored to the statistical and structural properties of its modality, enabling optimal feature extraction prior to multimodal integration.
MRI branch—3D Convolutional Neural Network (3D-CNN): MRI data are inherently volumetric, with spatially contiguous lesion patterns that cannot be captured by traditional tabular methods. We employed a 3D convolutional neural network consisting of three convolutional layers (filters: 8, 16, 32) with kernel size 3 × 3 × 3 and interleaved batch normalization and ReLU non-linearities to stabilize training and enhance feature representation [34]-[36]. Each block was followed by 3D max pooling (stride 2) to progressively reduce spatial resolution while preserving salient structural features. The final convolutional block output (size: 32 × 4 × 4 × 4) was flattened and projected into a 64-dimensional latent embedding, representing high-level structural descriptors of lesion morphology and distribution. This design is motivated by clinical evidence that lesion size, topology, and location critically influence post-stroke cognitive outcomes.
EEG branch—1D Convolutional Neural Network (1D-CNN): EEG is a temporal signal characterized by oscillatory dynamics across multiple frequency bands. To capture both short- and long-range dependencies in time, we implemented a 1D convolutional pipeline with two layers (filters: 16, 32; kernel sizes: 7 and 5). Convolutions were followed by batch normalization, ReLU, and adaptive average pooling, which compressed temporal sequences into a compact representation invariant to local phase shifts. The resulting flattened feature vector was mapped into a 32-dimensional embedding. This design captures frequency-specific EEG alterations, particularly enhanced delta–theta and attenuated alpha–beta rhythms, which are clinically associated with post-stroke cognitive slowing and impaired cortical function.
Clinical branch—Multilayer Perceptron (MLP): Clinical features are structured tabular variables with heterogeneous scales (continuous, ordinal, binary). To model these effectively, we employed a fully connected MLP with two hidden layers (64 and 32 units), each followed by ReLU activation and batch normalization to address feature scaling differences and stabilize training. The final projection produced a 16-dimensional embedding. This branch captures additive and interactive effects of risk factors such as age, NIHSS, education, and MoCA, which are widely recognized as key predictors of cognitive decline in stroke cohorts.
Fusion and Classification: The outputs from all three branches (64 + 32 + 16 = 112 features) were concatenated to form a unified multimodal representation. This fused vector was passed through a two-layer classifier (hidden: 64 → 32 units, with dropout = 0.2 for regularization) before a final sigmoid output layer, yielding the probability of cognitive decline.
Model Complexity and Training Considerations: The total architecture comprised approximately 198,000 trainable parameters, making it lightweight compared to conventional medical imaging models, yet expressive enough for multimodal integration. The relatively small parameter count was intentional, given the moderate dataset size (N = 800), to reduce overfitting risk. By aligning the architecture with the statistical properties of each modality (3D convolutions for MRI, 1D convolutions for EEG, MLP for tabular data), the model leverages domain-specific inductive biases while retaining flexibility through late fusion. This design reflects an important principle in clinical AI: architectures must not only maximize predictive performance but also maintain interpretability and computational feasibility for eventual clinical deployment [37]-[39].
2.3. Training and Optimization
The multimodal network was trained end-to-end using strategies designed to balance convergence speed, generalization ability, and robustness to class imbalance. All experiments were conducted in PyTorch, with GPU acceleration leveraged when available to enable efficient 3D convolutional computations. No explicit data augmentation was applied to MRI or EEG inputs in the current experiments. Regularization relied on architectural constraints (lightweight design), dropout (p = 0.2) in the fusion classifier, batch normalization, and AdamW weight decay. While this proved sufficient for proof-of-concept validation, future work will incorporate simple modality-specific augmentations, including random affine transformations and intensity jitter for MRI, and temporal jittering or frequency masking for EEG, to further enhance generalization.
Optimizer: We employed the AdamW optimizer (learning rate = 0.001, weight decay = 1 × 10−4). AdamW combines the adaptive learning-rate benefits of Adam with a decoupled weight decay formulation that improves generalization. This choice was particularly important for our dataset, which contained only 800 patients. Overfitting was a key risk, and AdamW’s ability to stabilize updates while penalizing overlarge weights allowed the model to converge smoothly without excessive variance [40]-[42].
Learning rate scheduling: To further enhance training stability, we used a cosine annealing scheduler over 8 epochs. The cosine decay gradually reduces the learning rate from its initial maximum to near-zero, enabling large exploratory updates in early epochs and fine-grained convergence in later epochs [43]. This cyclical pattern is advantageous in medical AI tasks where dataset size is limited, as it reduces the likelihood of the optimizer becoming trapped in sharp local minima that generalize poorly.
Batch size: A mini-batch size of 16 was selected to balance gradient estimation stability with GPU memory constraints. Small batch sizes introduce beneficial stochasticity in gradient updates, acting as an implicit regularizer [44]-[46]. This is especially valuable in synthetic datasets where the risk of memorization is elevated.
Loss function: The model was optimized with binary cross-entropy with logits (BCEWithLogitsLoss), which combines a sigmoid activation with binary cross-entropy loss in a numerically stable formulation. This objective directly models the probability of cognitive decline and penalizes misclassification according to cross-entropy divergence, a standard for binary clinical prediction tasks.
Device and computational considerations: All experiments were executed on a single GPU when available; otherwise, CPU fallback was used. The model comprised ~198k parameters, making it computationally lightweight compared to standard deep medical imaging networks, allowing efficient training even without specialized high-memory GPUs. Training for 8 epochs typically completed in under 15 minutes on a Tesla T4 GPU.
2.4. Evaluation Metrics
To comprehensively assess model performance, we employed a combination of discrimination metrics, diagnostic plots, interpretability tools, and representational analyses. This multi-layered evaluation ensured not only predictive accuracy but also clinical interpretability and methodological robustness.
Discrimination metrics
Model discrimination ability was quantified using:
Area Under the Receiver Operating Characteristic Curve (AUC): A global measure of the model’s ability to separate patients with vs without cognitive decline across all classification thresholds. AUC is widely used in clinical risk prediction due to its threshold-independent nature [47]-[49].
Average Precision (AP): A summary statistic of the Precision-Recall curve, particularly informative under class imbalance (positive prevalence ≈ 0.91). AP emphasizes performance on the minority (non-decline) class, which is clinically relevant since identifying preserved cognition is as important as detecting decline.
Accuracy: The proportion of correctly classified patients at a fixed threshold (0.5). While simple, accuracy was interpreted with caution due to class imbalance.
Diagnostic plots
We visualized model calibration and decision thresholds through:
ROC curves (sensitivity vs specificity trade-offs).
Precision-Recall (PR) curves to highlight positive predictive value across thresholds.
Calibration curves, plotting predicted vs observed probabilities, to evaluate whether risk estimates were well-aligned with actual outcomes - a crucial property for clinical deployment where mis calibrated probabilities can misguide treatment decisions.
Interpretability analyses
To open the “black box” and highlight clinically meaningful drivers:
Permutation importance (clinical features): Each variable was permuted independently to quantify the change in validation AUC. This method identifies which clinical attributes (e.g., baseline MoCA, age, NIHSS) most influenced predictions.
Gradient-based saliency maps: Input gradients were visualized for MRI volumes and EEG time-series, highlighting lesion voxels and temporal EEG segments most relevant for classification. This provided a mechanistic link between model predictions and domain-specific pathophysiology.
Representational analysis: To explore the internal representations learned by the multimodal fusion network, we projected fused latent embeddings into two dimensions using Uniform Manifold Approximation and Projection (UMAP). This visualization revealed whether patients clustered by outcome class, thereby validating that the network learned clinically meaningful separations rather than noise.
Finally, we conducted unimodal ablation experiments by training MRI-only, EEG-only, and clinical-only models. Comparison with the multimodal fusion architecture allowed us to quantify the contribution of each modality and to demonstrate the synergistic advantage of integrative modelling.
3. Results
The synthetic cohort size (N = 800) was selected to balance model expressiveness and overfitting risk for a network with approximately 198k parameters. Preliminary learning-curve analysis showed that validation AUC plateaued after ~600 samples, with diminishing returns beyond this point, suggesting that the chosen sample size was sufficient to train the proposed architecture without severe underfitting.
3.1. Performance Metrics
The multimodal deep learning model demonstrated promising predictive performance in classifying post-stroke cognitive decline. During training, the network achieved a best validation AUC of 0.7327 at epoch 3, indicating effective generalization before later epochs exhibited mild overfitting. This performance consistently exceeded that of unimodal baselines (MRI-only, EEG-only, and Clinical-only models), highlighting the value of multimodal integration. To quantify statistical uncertainty, 1,000 bootstrap resamples of the test set were used to estimate 95% confidence intervals for all metrics. The multimodal model achieved a test AUC of 0.73 (95% CI: 0.69 - 0.77), average precision of 0.92 (95% CI: 0.89 - 0.95), and accuracy of 0.84 (95% CI: 0.80 - 0.88). Confidence intervals confirm that performance gains over unimodal baselines are robust and unlikely to arise from sampling variability.
On the independent test set, the fusion model maintained stable performance, as reflected in both the Receiver Operating Characteristic (ROC) and Precision–Recall (PR) curves. The ROC curve (Figure 1) showed a favourable trade-off between sensitivity and specificity across thresholds, while the PR curve (Figure 2) confirmed robustness in handling class imbalance (positive prevalence ≈ 0.91). Model calibration was further assessed through calibration curves (Figure 3). Although the overall shape approximated the ideal diagonal line, the model demonstrated moderate misalignment, with a tendency toward overconfidence at higher predicted probabilities. This suggests that while discrimination was acceptable, future work should incorporate post-hoc calibration techniques (e.g., isotonic regression, Platt scaling) before clinical application.
The curve illustrates sensitivity-specificity trade-offs across probability thresholds, with the area under the curve (AUC) reflecting overall discrimination.
Figure 1. Receiver Operating Characteristic (ROC) curve for the multimodal model on the test set.
Figure 2. Precision-Recall (PR) curve for the multimodal model on the test set.
The plot demonstrates positive predictive value across varying recall levels, highlighting robustness under class imbalance.
Figure 3. Calibration curve for the multimodal model.
The plot compares mean predicted probabilities with observed outcome frequencies. While the model showed approximate alignment, deviations at higher probabilities indicated slight overconfidence.
3.2. Clinical Feature Importance
To interpret the role of non-imaging predictors, we applied permutation importance analysis on the clinical feature branch. Results revealed that baseline MoCA, age, and years of education emerged as the most influential variables (Figure 4). This pattern aligns with established clinical evidence: baseline MoCA reflects pre-stroke cognitive status, while both age and educational attainment are proxies for cognitive reserve, factors consistently associated with long-term neurocognitive resilience. Other contributors included stroke severity (NIHSS) and vascular comorbidities such as hypertension and diabetes, albeit with lower relative importance. These findings underscore the complementary role of structured clinical data in enhancing model interpretability and providing insight into patient-specific risk profiles. By confirming that the model assigns weight to clinically meaningful variables, permutation importance increases trustworthiness and face validity, reinforcing the potential translational utility of the framework.
![]()
Figure 4. Permutation importance of clinical features.
Box plots display the distribution of validation AUC changes when individual features were permuted. Baseline MoCA, age, and education were the most impactful contributors, aligning with clinical knowledge on cognitive reserve and stroke recovery.
3.3. Saliency Analyses
To examine the spatial and temporal focus of the multimodal network, we generated gradient-based saliency maps for MRI and EEG modalities. These analyses provided insight into whether the model’s decision-making aligned with biologically plausible features.
For the MRI branch, saliency consistently emphasized lesion-like regions within the synthetic brain volumes (Figure 5, Figure 6). This confirms that the 3D-CNN branch was not attending to background noise but rather to lesion patterns that were explicitly linked to outcome probabilities during dataset generation. Such focus reflects clinically meaningful model behaviour, as lesion burden and location are known determinants of post-stroke cognitive outcomes. For the EEG branch, saliency analysis revealed that the network preferentially weighted delta–theta enriched segments while down-weighting higher-frequency alpha–beta regions (Figure 7). This pattern is consistent with electrophysiological markers of cognitive slowing and cortical dysfunction after stroke, further validating the biological plausibility of the model’s learned representations. Overall, saliency analyses reinforce that the model’s predictive performance was driven by domain-relevant imaging and electrophysiological features, enhancing transparency and supporting potential clinical trust.
![]()
Figure 5. MRI saliency maps highlighting lesion regions in an impaired patient.
Bright regions indicate voxels with the greatest influence on prediction.
Figure 6. MRI saliency maps for a preserved patient.
Minimal lesion-related activity is emphasized, consistent with lower predicted risk.
Figure 7. EEG saliency visualization.
Gradient-based importance scores highlight delta–theta segments as key drivers of impaired predictions, consistent with cortical slowing in post-stroke patients.
3.4. Embedding Visualization and Contribution Analysis
To better understand how the network organized multimodal information, we examined the fused latent representations using Uniform Manifold Approximation and Projection (UMAP). The two-dimensional projection (Figure 8) revealed clear clustering of decline vs non-decline patients, with impaired patients forming a dense cluster and preserved patients occupying a distinct, though smaller, region. This indicates that the network successfully learned discriminative embeddings that separate outcome classes in a low-dimensional space, validating the effectiveness of the fusion layer in capturing cross-modal synergies.
We further quantified the added value of multimodal integration through contribution analysis. Specifically, we compared the multimodal fusion model against unimodal baselines trained independently on MRI, EEG, and clinical data. Results (Figure 9) demonstrated that fusion improved discrimination by approximately 10% - 15% in AUC relative to the strongest unimodal branch. This finding highlights the synergistic benefit of multimodal modelling: while unimodal models captured domain-specific risk factors, only the integrative framework leveraged complementary information across imaging, electrophysiology, and clinical profiles to achieve superior performance. These analyses collectively support the premise that multimodal deep learning is not merely aggregative but additively informative, yielding representations and predictive power unattainable by unimodal models alone.
Figure 8. UMAP embedding of fused latent representations.
The two-dimensional projection demonstrates clear clustering of decline (red) vs non-decline (blue) patients, indicating that the model learned outcome-separating features in its latent space.
Figure 9. Contribution analysis across unimodal and multimodal models.
Bar plots of AUC demonstrate that the multimodal fusion model outperforms unimodal MRI, EEG, and clinical branches by ~10% - 15%, highlighting the synergistic advantage of integrative modelling.
3.5. Modal Contribution and Representation Synergy
To further disentangle the relative contributions of individual modalities, we performed complementary analyses examining validation performance, embedding structure, and proportional contributions. First, unimodal baselines were benchmarked against the multimodal fusion model (Figure 10). While MRI and clinical features achieved moderate discrimination individually (AUC ≈ 0.72 and 0.66, respectively), EEG alone underperformed (AUC ≈ 0.55). Importantly, the fusion model outperformed all unimodal models, indicating that multimodal integration captures complementary information does not present in single domains. Second, the UMAP visualization of fused embeddings (Figure 11) confirmed that the representational structure was not solely driven by a single modality. Patients with cognitive decline formed multiple tightly packed subclusters, while preserved patients separated distinctly, suggesting that the network encoded shared but heterogeneous decline trajectories across modalities.
Finally, relative contribution analysis quantified the additive value of each modality to the fused AUC (Figure 12). MRI contributed the largest share (≈37%), followed by clinical features (≈34%), and EEG (≈28%). These proportions underscore that although MRI provided the strongest single-modality signal, clinical and EEG data contributed substantial, non-redundant information that improved model discrimination. Together, these results highlight the synergistic advantage of multimodal fusion. By leveraging complementary signals from structural lesions, electrophysiological slowing, and clinical context, the model achieved superior predictive performance, supporting the translational relevance of integrative modelling in post-stroke cognitive prognosis.
Figure 10. Validation AUC across unimodal and multimodal models.
The multimodal fusion model outperformed unimodal MRI, EEG, and clinical models, confirming the added value of cross-domain integration.
Figure 11. UMAP embedding of fused representations.
Patients with cognitive decline form distinct clusters compared to preserved patients, reflecting effective outcome separation in the latent space.
Figure 12. Relative contributions of MRI, EEG, and clinical features to fused AUC.
Pie chart in Figure 12, illustrates that all three modalities provide additive, non-redundant value, with MRI contributing most strongly.
4. Discussion
This proof-of-concept demonstrates that multimodal deep learning combining structural MRI, EEG time-series, and structured clinical data can meaningfully predict post-stroke cognitive decline (PSCD) even in a constrained setting. The fusion model reached a best validation AUC = 0.7327 (epoch 3) and consistently exceeded all unimodal baselines, indicating that the modalities contribute complementary rather than redundant information. Below we interpret the results, examine robustness, and outline what is needed for clinical translation. This class ratio inflates baseline accuracy and necessitates cautious interpretation of threshold-dependent metrics. Accordingly, precision recall curves and calibration analyses were prioritized, and future studies will explore more balanced and population-representative prevalence settings.
4.1. What the Model Learned and Why It Matters
Multimodal superiority: The ablations and contribution plots show that MRI carries the largest single signal (lesion burden/topology), while clinical variables (baseline MoCA, age, education, NIHSS) and EEG (delta–theta dominance) add non-overlapping signal. The ≈10% - 15% AUC gain of fusion over the best unimodal branch suggests true synergy across modalities.
Biological plausibility: Saliency maps concentrated on lesion-like voxels in MRI and low-frequency EEG segments patterns consistent with post-stroke cortical slowing. Permutation importance elevated baseline MoCA, age, education, aligning with cognitive reserve theory and known prognostic factors. These convergences increase face validity and clinical trust.
Structure in representation space: UMAP of fused embeddings showed separable clusters for decline vs. non-decline, implying the model captured latent structure related to PSCD rather than memorizing noise. Multiple sub-clusters within the decline group hint at heterogeneous pathways to impairment (e.g., large-lesion vs. high-vascular-risk phenotypes), a promising direction for phenotyping.
4.2. Reliability, Calibration, and Clinical Usability
Discrimination vs. decision-making: ROC/PR curves are acceptable for a first pass, but clinical deployment needs more than discrimination [50] [51]. The calibration curve shows over-confidence at high predicted probabilities; before risk-stratified decisions (e.g., triaging intensive cognitive rehab), apply post-hoc calibration (temperature scaling, isotonic regression) and report Brier score.
Threshold selection: Because the class prevalence is high (positives ≈ 0.91), accuracy can mislead. Thresholds should be set using Youden’s J, cost-sensitive analysis, or Decision Curve Analysis (DCA) to maximize net clinical benefit given local costs of false positives/negatives.
4.3. Limitations and Risks of Bias
Synthetic data: Although designed to mirror clinical regularities, synthetic data cannot reproduce the full distributional complexity of real stroke cohorts (scanner/site effects, artifacts, comorbidity interactions). The positive rate (0.91) induces severe imbalance, which can inflate PR curves and bias thresholds.
Potential shortcut learning: Even with saliency checks, gradient methods can be noisy and miss spurious cues. Add occlusion/RISE, integrated gradients, or 3D Grad-CAM for MRI and spectral occlusion for EEG to triangulate explanations.
Generalization risk: No external validation was performed. Real-world performance will drop under domain shift (different hospitals, scanners, EEG montages, populations).
4.4. What’s Needed for Clinical Translation
Data & validation.
Curate multi-center cohorts with harmonized MRI (e.g., bias-field correction, ComBat), artifact-clean EEG (ICA), and standardized MoCA/NIHSS.
Use nested cross-validation, site-stratified splits, and an external test set; report CIs (bootstrap) and perform DeLong tests for AUC differences.
Modelling improvements.
Address imbalance with class-weighted BCE, focal loss, or balanced mini-batches.
Add data augmentation: MRI (random affine/intensity jitter), EEG (jitter/time-warp/mixup).
Explore self-supervised pretraining (MAE/SimCLR for MRI; contrastive/temporal masking for EEG) and cross-modal transformers/GNNs for connectomic features (DTI/fMRI when available).
Quantify uncertainty with MC-dropout or deep ensembles and surface it alongside risk.
Clinical outputs.
Calibrate probabilities and convert to actionable risk strata (e.g., <20% low, 20% - 50% intermediate, >50% high).
Provide case-level reports: salient MRI slices, EEG segments, top clinical contributors, and a short textual explanation to support shared decision-making.
Governance & reporting.
Follow TRIPOD-AI/PROBAST-AI for transparent reporting and bias assessment; plan for prospective silent-mode validation before any interventional use.
Use federated learning to train across hospitals without sharing raw data; monitor calibration drift post-deployment.
4.5. Overall Interpretation
Despite its synthetic nature and moderate discrimination (AUC ≈ 0.73), the system exhibits the three pillars required for clinical AI promise: 1) added value of multimodal fusion over single modalities, 2) plausible mechanisms evidenced by importance and saliency analyses, and 3) organized latent structure consistent with clinical heterogeneity. With rigorous external validation, proper calibration, and workflow-aligned reporting, this approach could evolve into a decision-support tool to prioritize neuropsychological assessment, tailor rehabilitation intensity, and plan caregiver resources early after stroke. Future work will therefore focus on: 1) multi-center external validation with harmonization; 2) longitudinal/time-to-event modelling (e.g., DeepSurvy/transformer-based dynamic risk to predict when decline occurs); 3) privacy-preserving federated training; and 4) deployment-grade calibration and uncertainty quantification to ensure trustworthy bedside use.
5. Conclusion
This study presents a proof-of-concept multimodal deep learning framework for predicting post-stroke cognitive decline (PSCD), integrating structural MRI, EEG time-series, and clinical features within a unified pipeline. The model achieved promising discrimination (AUC ≈ 0.73) and demonstrated interpretability through permutation importance and saliency analyses, which highlighted clinically plausible predictors such as baseline MoCA, lesion burden, and EEG slowing. Furthermore, embedding visualizations revealed meaningful latent structures that separated decline from non-decline patients, underscoring the potential of representation learning to capture underlying heterogeneity in stroke outcomes. Although the present work relies on synthetic data, the methodology establishes a scalable and transparent framework that can be directly adapted to real-world, multi-center cohorts. By confirming that multimodal fusion consistently outperforms unimodal models, this study provides evidence that structural, functional, and clinical modalities are additively informative rather than redundant. Ultimately, the findings support the clinical promise of multimodal AI as a foundation for precision rehabilitation strategies. With rigorous external validation, integration of longitudinal trajectories, and privacy-preserving deployment strategies such as federated learning, such models may enable early risk stratification, optimized rehabilitation intensity, and proactive caregiver planning. This work therefore represents a critical first step toward clinically deployable, interpretable, and personalized prediction of post-stroke cognitive decline.