Abstract

The deployment of artificial intelligence models in psychiatric care faces a critical challenge: models trained in one healthcare institution often fail to generalize across different populations, clinical practices, and healthcare systems. This study investigates domain adaptation and robustness techniques to ensure psychiatric AI models maintain predictive accuracy when deployed across diverse institutional settings. We developed a comprehensive framework integrating transfer learning, adversarial domain adaptation, and federated learning approaches to address cross-institutional heterogeneity. Using synthetic clinical datasets representing six distinct hospitals with varying patient demographics, treatment protocols, and documentation practices, we evaluated model performance under realistic non-independent and identically distributed conditions. Our domain adaptation approach achieved mean AUC-ROC of 0.847 (95% CI: 0.815 - 0.879) across target institutions, representing a 9.1% improvement over direct model deployment without adaptation. Feature importance analysis identified age, depression severity scores, prior hospitalizations, and medication adherence as domain-invariant predictors, while hospital type and documentation style emerged as domain-specific confounders. Risk stratification analysis demonstrated consistent performance across institutions, with observed readmission rates ranging from 7.5% - 8.9% in the lowest risk category to 79.5% - 83.8% in the highest risk category. These findings establish that domain adaptation techniques can effectively mitigate institutional heterogeneity, enabling reliable deployment of psychiatric AI models across diverse healthcare settings while maintaining predictive accuracy and clinical utility.

Keywords

Domain Adaptation, Transfer Learning, Psychiatric Ai, Cross-Institutional Generalization, Federated Learning, Machine Learning Robustness, Precision Psychiatry, Clinical Decision Support

Share and Cite:

1. Introduction

Artificial intelligence and machine learning have demonstrated substantial potential for improving psychiatric care through predictive modelling, diagnostic assistance, and treatment optimization [1]. Recent advances in deep learning and ensemble methods have enabled the development of sophisticated models capable of predicting clinical outcomes such as hospital readmission, treatment response, and suicide risk with accuracies approaching or exceeding traditional clinical assessment [2]. However, a critical barrier to widespread clinical deployment is the challenge of cross-institutional generalization: models trained in one healthcare setting often exhibit significant performance degradation when applied to patient populations from different institutions [3].

The generalization challenge in psychiatric AI stems from multiple sources of institutional heterogeneity. Patient populations differ across hospitals in demographic composition, socioeconomic status, illness severity, and comorbidity profiles [4]. Clinical practices vary in treatment protocols, medication preferences, and discharge planning procedures. Documentation practices differ in coding conventions, assessment instruments, and electronic health record structures [5]. These variations create domain shifts that can substantially degrade model performance when models are deployed outside their training environment [6].

Domain adaptation techniques offer a promising approach to addressing cross-institutional heterogeneity [7]. These methods aim to learn representations that are invariant across domains while preserving predictive power for the target task. Transfer learning approaches leverage knowledge from a source domain to improve learning in a related target domain [8]. Adversarial training methods learn domain-invariant features by training a classifier to distinguish between domains while the feature extractor learns to confuse it [9]. Federated learning enables collaborative model training across institutions without centralizing sensitive patient data [10].

Figure 1 illustrates our proposed cross-institutional domain adaptation framework for psychiatric AI models. The framework integrates multiple adaptation strategies to enable model generalization from a source hospital to diverse target institutions while maintaining predictive accuracy and clinical utility.

Figure 1. Cross-Institutional domain adaptation framework. The framework integrates transfer learning, adversarial training, and feature alignment to enable model generalization from a source hospital to diverse target institutions while maintaining predictive accuracy.

This study makes the following key contributions:

1) We develop a comprehensive domain adaptation framework specifically designed for psychiatric AI applications, integrating transfer learning, adversarial training, and federated learning approaches.

2) We characterize the sources and impact of institutional heterogeneity on psychiatric prediction models using realistic synthetic datasets representing diverse hospital settings.

3) We identify domain-invariant features that maintain predictive power across institutions versus domain-specific confounders that contribute to performance degradation.

4) We demonstrate that domain adaptation techniques can achieve centralized-level performance across diverse target institutions while preserving data privacy and regulatory compliance.

5) We provide practical guidance for implementing cross-institutional AI deployment in psychiatric care settings.

2. Related Work

2.1. Domain Adaptation in Healthcare AI

Domain adaptation has emerged as a critical area in machine learning research, addressing the problem of training models on data from a source domain that must perform well on a different but related target domain [11]. In healthcare applications, domain adaptation techniques have been applied to medical imaging, electronic health record analysis, and clinical prediction tasks [12]. Early approaches focused on feature-based methods that transform data to minimize distribution differences between domains [13]. More recent work has explored deep learning approaches that learn domain-invariant representations through adversarial training or discrepancy-based methods [14].

2.2. Cross-Institutional Generalization Challenges

The challenge of model generalization across healthcare institutions has been documented across multiple medical specialties [15]. Studies have shown that machine learning models for sepsis prediction, mortality risk stratification, and length-of-stay estimation exhibit substantial performance degradation when transferred between hospitals [16]. These failures have been attributed to differences in patient populations, clinical practices, data collection procedures, and documentation conventions [17]. The psychiatric context presents unique challenges due to the subjective nature of symptom assessment, variability in diagnostic practices, and the influence of cultural and social factors on illness presentation [18].

2.3. Federated Learning for Privacy-Preserving Collaboration

Federated learning has emerged as a promising paradigm for training machine learning models across decentralized data sources without sharing raw patient data [19]. In this approach, model parameters are shared and aggregated centrally while training data remains at each institution [20]. This approach addresses privacy regulations such as HIPAA and GDPR while enabling collaborative model development on larger, more diverse datasets [21]. Recent work has explored federated learning for healthcare applications including medical imaging, clinical prediction, and drug discovery [22].

3. Methods

3.1. Study Design and Data Sources

We conducted a simulation study modelling six distinct hospitals with heterogeneous psychiatric patient populations. The source hospital represented a large academic medical center with comprehensive psychiatric services. Five target hospitals represented diverse clinical settings: a community health center serving underserved urban populations, a veteran’s affairs hospital, a private psychiatric institute, a rural regional hospital, and a university-affiliated tertiary care center [23].

3.2. Synthetic Data Generation

Patient data were synthesized using a multivariate Gaussian framework. Each patient was represented by 18 clinical features drawn from a multivariate normal distribution N(μ_s, Σ_s), where μ_s and Σ_s are site-specific mean vectors and covariance matrices calibrated to published psychiatric cohort statistics [24]-[26]. Covariance structure was specified to reflect known clinical correlations: PHQ-9 and GAD-7 scores were correlated at r = 0.62; prior hospitalizations and illness duration at r = 0.54; medication adherence and readmission probability at r = −0.48. The 30-day readmission label was generated via a logistic function of a weighted linear combination of the 18 features, with coefficients derived from published readmission risk literature [18] [19]: P(readmission) = σ(β₀ + β₁∙PHQ9 + β₂·priorAdmissions + β₃·adherence + ε), where ε~N (0, 0.1) introduces label noise. Site-specific distribution shifts were introduced along three axes: 1) covariate shift μ_s was shifted by a site-specific offset vector drawn from U (−1.5, 1.5) per feature; 2) missingness each target site had a randomly assigned missing-at-random rate of 5% - 20% per feature, imputed using multivariate mean imputation computed from source training data; 3) label noise target sites received an additional Bernoulli noise term on readmission labels with site-specific flip probability drawn from U (0.02, 0.08), simulating diagnostic coding variability. Source hospital: n = 2500. Target hospitals A-E: n = 500, 600, 650, 750, 800 respectively, reflecting realistic multi-site recruitment asymmetry.

3.3. Domain Adaptation Approaches

We evaluated three domain adaptation approaches: 1) Transfer learning with fine-tuning, where a base model trained on source data is adapted to target data through continued training with reduced learning rates [27]; 2) Adversarial domain adaptation, which trains a domain discriminator to distinguish between source and target features while the feature extractor learns to produce domain-invariant representations [28]; and 3) Federated learning with domain aggregation, which trains a global model across all institutions while preserving data locality [29].

Site Proxy Variables at Deployment: Variables including hospital type, treatment protocol, documentation style, and geographic region (identified as domain-specific confounders in Section 4.3) were included as model inputs during source training but were explicitly excluded from the final prediction head at deployment. This separation was implemented to prevent the model from exploiting site identity as a shortcut predictor, which would inflate apparent adaptation performance and reduce real-world portability. At deployment, only the 12 domain-invariant clinical features (PHQ-9, GAD-7, prior hospitalizations, medication adherence, age, illness duration, gender, socioeconomic status, substance use, social support, trauma history, diagnosis) were passed to the prediction head. The six site-proxy features were used only by the domain discriminator during adversarial training and discarded thereafter. Target Data Availability per Adaptation Method. The three evaluated methods differed in what target-site data were available during adaptation. 1) Transfer learning with fine-tuning: required a small, labelled target set (n = 50 - 100 labelled patients per site), used exclusively for continued gradient updates on the pre-trained source model. 2) Adversarial domain adaptation: used unlabelled target data only during domain discriminator training; labels from the target site were withheld during feature-extractor training to simulate an unsupervised adaptation setting. 3) Federated learning: each target site contributed both labelled local data and participated in federated averaging rounds; no raw data left the local site. In all cases, adaptation data were drawn from a held-out partition of each target hospital’s dataset, separate from the test set used for final evaluation.

3.4. Model Architecture and Training

The base predictive model was a gradient boosting classifier (GBM; 200 estimators, max depth 5, learning rate 0.1), selected for its strong performance with tabular clinical data. This GBM served as the prediction head in all three adaptation pipelines. For transfer learning, the GBM was re-fitted on target labeled data using source-trained leaf weights as initialization via warm-starting. For adversarial domain adaptation, the GBM feature layer was replaced with a two-layer MLP encoder (256 → 128 units, ReLU activations), producing a 128-dimensional embedding fed to both the GBM prediction head and a gradient-reversal domain discriminator (two fully connected layers, sigmoid output); the entire system was trained end-to-end using gradient reversal, with convergence assessed over 50 epochs using early stopping on validation AUC. For federated learning, a global GBM was aggregated using FedAvg across all six sites over 20 communication rounds [30]. The gradient boosting base model was not used standalone for adversarial training; the MLP encoder described above was the feature extractor in that pipeline.

3.5. Evaluation Metrics

Primary performance metrics included area under the receiver operating characteristic curve (AUC-ROC), accuracy, precision, recall, and F1-score [31]. Calibration was assessed using Brier score and calibration plots [32]. Cross-institutional consistency was evaluated using correlation of performance metrics and prediction distributions across hospitals. Feature importance was quantified using SHAP values to identify domain-invariant versus domain-specific predictors [33]. All reported mean performance values and 95% confidence intervals were computed over 20 independent simulation replicates with different random seeds (seeds 1 - 20), each producing a fresh synthetic dataset, train/test partition, and model initialization. CIs were derived as the 2.5th and 97.5th percentiles of the replicate distribution. The train/test split was 80/20 within each hospital, stratified by readmission label. Hyperparameter selection (number of estimators, max depth, learning rate, MLP hidden size, adversarial loss weight λ) was performed by grid search on a held-out 10% validation partition from the source hospital only, prior to any target-site evaluation.

4. Results

4.1. Performance Comparison Across Institutions

The source model achieved AUC-ROC of 0.891 (95% CI: 0.872 - 0.910) on held-out test data from the source hospital. When directly deployed to target hospitals without adaptation, performance degraded substantially with mean AUC of 0.756 (95% CI: 0.728 - 0.784) across target institutions. Domain adaptation techniques recovered much of this performance loss, achieving mean AUC of 0.847 (95% CI: 0.815 - 0.879) across target institutions, representing a 9.1% improvement over direct deployment without adaptation.

Figure 2 presents the performance comparison across healthcare institutions. The domain adaptation approach achieved clinically acceptable performance (AUC greater than 0.80) at all target hospitals, whereas direct deployment without adaptation fell below the clinical utility threshold at three of five target institutions. Table 1 presents detailed performance metrics across all institutions.

Figure 2. Model performance across healthcare institutions. Comparison of AUC-ROC and accuracy for source hospital, target hospitals without adaptation, and with domain adaptation. Error bars represent 95% confidence intervals. The dashed line indicates the clinical utility threshold (AUC = 0.80).

Table 1. Full per-institution performance metrics across all baselines and adaptation methods. Values are means over 20 simulation replicates; 95% CIs omitted from table for readability but reported in text.

Across all five target hospitals, adversarial domain adaptation consistently outperformed both no-adaptation and transfer learning baselines on AUC-ROC, accuracy, and Brier score. Federated learning ranked second overall, achieving AUC values within 0.01 of adversarial DA while requiring no labelled target data centralization. The largest absolute gain from adaptation was observed at Target B (ΔAUC = +0.082) and the smallest at Target E (ΔAUC = +0.085), indicating consistent benefit across heterogeneous site profiles.

4.2. ROC Analysis

ROC curves demonstrated excellent discrimination for the domain adaptation approach, with AUC values significantly exceeding the 0.80 threshold for clinical utility across all target hospitals [34]. Figure 3 presents the ROC curves comparing the source model, target deployment without adaptation, domain adaptation, and federated learning approaches. The source model without adaptation showed inferior performance, particularly at low false positive rates where clinical utility is maximized.

4.3. Feature Importance Analysis

Feature importance analysis revealed distinct patterns for domain-invariant versus domain-specific features. Figure 4 presents the SHAP-based feature importance analysis. Clinical variables including PHQ-9 depression scores, prior hospitalizations, medication adherence, age, illness duration, and GAD-7 anxiety scores showed consistent importance across all institutions, identifying them as domain-invariant predictors. In contrast, institutional variables such as hospital type, treatment protocol, insurance status, documentation style, geographic region, and staff training showed variable importance across sites, indicating their role as domain-specific confounders [35].

Figure 3. Receiver operating characteristic curves for cross-institutional generalization. Comparison of source model, target deployment without adaptation, domain adaptation, and federated learning approaches. The optimal operating point (sensitivity = 0.79, specificity = 0.85) is indicated by the red star.

Figure 4. Feature importance analysis using SHAP values. Domain-invariant features (left) maintain consistent predictive power across institutions with lower variability, while domain-specific features (right) show higher variability across institutions.

4.4. Calibration Analysis

Calibration plots assessed the reliability of predicted probabilities for clinical decision-making. Figure 5 presents the calibration curves for the source hospital, target hospital without adaptation, and target hospital with domain adaptation. The domain adaptation approach demonstrated well-calibrated probability estimates across all target hospitals, with predicted probabilities closely matching observed frequencies (Brier score = 0.156). Direct deployment without adaptation showed systematic miscalibration (Brier score = 0.198), with overestimation of risk at lower probability ranges and underestimation at higher ranges [36].

Figure 5. Calibration plot for cross-institutional model performance. Perfect calibration indicated by the diagonal dashed line. The domain adaptation approach achieves well-calibrated predictions comparable to the source hospital, while deployment without adaptation shows systematic miscalibration.

4.5. Confusion Matrix Analysis

Confusion matrices revealed the impact of domain adaptation on classification accuracy. Figure 6 presents the confusion matrices for the source hospital, target hospital without adaptation, and target hospital with domain adaptation. Without adaptation, target hospitals showed increased false positives and false negatives compared to source performance. Domain adaptation reduced both error types, approaching source-level performance. The largest improvements were observed in reducing false negatives, which are particularly critical for patient safety in readmission prediction [37].

Figure 6. Confusion matrices showing impact of domain adaptation. Classification performance at source hospital (left), target without adaptation showing degraded performance (center), and target with domain adaptation showing recovered performance (right).

4.6. Learning Curve Analysis

Learning curves demonstrated the convergence behaviour of domain adaptation over training epochs. Figure 7 presents the learning curves for different approaches. The domain adaptation approach showed steady improvement during the first 12 epochs, converging to stable performance by epoch 35. Without adaptation, target performance remained suboptimal throughout training, indicating that additional training data from the source domain alone cannot overcome institutional heterogeneity [38].

Figure 7. Learning curve showing domain adaptation convergence. Performance improvement over training epochs demonstrating that domain adaptation converges to near-source performance while training without adaptation plateaus at lower performance levels.

4.7. Risk Stratification Consistency

Risk stratification analysis evaluated the consistency of predicted risk categories across institutions. Figure 8 presents the observed readmission rates by predicted risk category across hospitals. Patients were stratified into five risk categories based on predicted readmission probability. Observed readmission rates showed consistent patterns across hospitals, with the lowest risk category showing 7.5% - 8.9% readmission rates and the highest risk category showing 79.5% - 83.8% rates. This 10-fold risk gradient supports the clinical utility of the domain adaptation approach for identifying high-risk patients regardless of institutional setting [39].

Figure 8. Risk stratification consistency across institutions. Observed readmission rates by predicted risk category demonstrate consistent performance across source and target hospitals, supporting clinical utility for identifying high-risk patients.

5. Discussion

5.1. Principal Findings

This study demonstrates that domain adaptation techniques can effectively address cross-institutional heterogeneity in psychiatric AI models. The key finding is that domain adaptation achieved centralized-level performance across diverse target institutions, with mean AUC-ROC of 0.847 compared to 0.756 for direct deployment without adaptation. This 9.1% improvement represents a clinically meaningful enhancement in predictive accuracy that could translate to improved patient outcomes through better risk stratification [40].

The identification of domain-invariant features provides important insights for model development. Clinical variables such as PHQ-9 depression scores, prior hospitalizations, medication adherence, age, illness duration, and GAD-7 anxiety scores maintained consistent predictive power across institutions, suggesting they capture fundamental biological and behavioural risk factors. In contrast, institutional variables such as hospital type, treatment protocol, and documentation style contributed to performance degradation when models were transferred across settings [41].

5.2. Clinical Implications

These findings have several important implications for clinical deployment of psychiatric AI models. First, domain adaptation should be considered a standard component of multi-site model deployment, particularly when target institutions differ substantially from the training environment. Second, models should prioritize domain-invariant clinical features over institution-specific variables to maximize generalizability [42]. Third, calibration assessment should be performed at each deployment site to ensure reliable probability estimates for clinical decision-making.

The risk stratification consistency observed across institutions supports the clinical utility of domain-adapted models for identifying high-risk patients who may benefit from enhanced discharge planning, intensive case management, or transitional care interventions [43]. The ability to reliably identify patients in the highest risk category, with observed readmission rates exceeding 80%, enables targeted resource allocation to those most likely to benefit.

5.3. Comparison with Prior Work

Our findings align with and extend prior research on domain adaptation in healthcare AI. Previous studies in medical imaging and general medicine have documented performance degradation with cross-institutional deployment and demonstrated benefits from domain adaptation techniques [44]. Our work extends this literature to psychiatric applications, where heterogeneity in symptom assessment and diagnostic practices creates unique challenges. The magnitude of improvement we observed with domain adaptation is consistent with prior healthcare studies, which have reported AUC improvements of 3-10% with various adaptation approaches [45].

5.4. Limitations

Several limitations should be considered when interpreting these findings. First, the use of synthetic data, while necessary for controlled evaluation, cannot fully replicate the complexity of real clinical data [46]. Second, our evaluation focused on a single prediction task (readmission risk) and may not generalize to other psychiatric outcomes. Third, we did not evaluate all possible domain adaptation techniques, and alternative approaches may achieve different performance. Fourth, the binary readmission outcome does not capture the full spectrum of post-discharge clinical trajectories.

5.5. Future Directions

Future research should prioritize validation in real-world clinical datasets with actual multi-site implementation. Integration of additional data modalities such as neuroimaging, genetic markers, and digital phenotyping may enhance prediction accuracy and generalizability. Development of adaptive models that continuously learn from new institutional data could further improve performance over time. Investigation of fairness and equity across demographic subgroups within and across institutions is essential to ensure equitable model performance.

6. Conclusions

This study establishes that domain adaptation techniques can effectively enable cross-institutional generalization of psychiatric AI models. Our comprehensive framework integrating transfer learning, adversarial training, and federated learning achieved centralized-level performance across diverse target hospitals, representing a 9.1% improvement over direct deployment without adaptation. The identification of domain-invariant clinical features versus domain-specific confounders provides practical guidance for model development and deployment.

The consistent risk stratification performance across institutions supports the clinical utility of domain-adapted models for identifying high-risk patients who may benefit from targeted interventions. These findings address a critical barrier to widespread clinical deployment of psychiatric AI and establish a foundation for multi-site implementation while maintaining data privacy and regulatory compliance. Future work should focus on real-world validation, integration of multimodal data, and continuous adaptive learning to further enhance generalizability and clinical impact.

Conflicts of Interest

The authors declare no conflicts of interest.

References