To evaluate the proposed Enhanced Multimodal Transformer (EMT) in a controlled setting, we generated a synthetic but clinically inspired multimodal dataset comprising functional magnetic resonance imaging (fMRI), genomic single-nucleotide polymorphism (SNP) profiles, and structured clinical variables. The dataset was constructed to mimic biologically plausible differences between Treatment-Resistant Depression (TRD) and non-TRD individuals, enabling the model to learn cross-modal patterns under balanced class conditions. Mean activation offsets were chosen to reflect effect directions consistently reported in neuroimaging studies of depression, with amygdala hyperactivation and prefrontal hypoactivation commonly associated with affective dysregulation and impaired cognitive control.

Functional MRI (fMRI) Data: Each fMRI sample was represented as a four-dimensional array with spatial dimensions of 48 × 48 × 24 voxels and 80 temporal frames. Gaussian noise (μ = 0, σ = 0.08) was used as a baseline activation level. Region-specific signal alterations were applied to emulate TRD-related neuropathology:

Amygdala hyperactivation was induced in TRD cases by adding a mean signal offset (μ = 0.5, σ = 0.1) in the voxel range corresponding to the right amygdala.Prefrontal hypoactivation in TRD was simulated by subtracting an offset (μ = 0.4, σ = 0.1) in the dorsolateral prefrontal region.Anterior cingulate activation was enhanced in non-TRD cases (μ = 0.3, σ = 0.1), reflecting better emotional regulation.

Genomic SNP Data: Each subject was assigned a vector of 800 SNPs with genotype values encoded as 0 (homozygous reference), 1 (heterozygous), or 2 (homozygous alternate). To embed predictive genetic markers, the first 100 SNPs were generated with distinct probability distributions between TRD and non-TRD groups. TRD samples had a higher frequency of alternate alleles (p = [0.4, 0.35, 0.25]), whereas non-TRD samples had predominantly reference alleles (p = [0.8, 0.15, 0.05]).

Clinical Features: Fifteen continuous clinical variables were drawn from a normal distribution (μ = 0, σ = 0.8). TRD cases were characterized by increased depression severity (+2.0), longer illness duration (+1.5), greater episode count (+1.2), reduced functional capacity (−1.5), and elevated anxiety comorbidity (+1.0).

The final synthetic cohort comprised N = 1000 subjects, evenly balanced between Treatment-Resistant Depression (TRD; n = 500) and non-TRD (n = 500) groups. Subjects were randomly partitioned into training (70%), validation (15%), and test (15%) sets at the subject level to avoid data leakage across modalities. All reported results correspond to performance on the held-out test set, with validation data used exclusively for early stopping and learning-rate scheduling. The dataset was balanced with an equal number of TRD and non-TRD samples to prevent class imbalance bias.

The Enhanced Multimodal Transformer (EMT) integrates three heterogeneous data modalities-fMRI, genomic SNPs, and clinical features within a unified Transformer-based architecture. The model design incorporates modality-specific encoders, positional embeddings, and dedicated modality tokens to facilitate cross-domain representation learning. The fMRI patch size (6 × 6 × 4 voxels) was selected to balance spatial resolution with computational tractability, consistent with prior work on patch-based neuroimaging Transformers. The hidden dimension (d_model = 256) and Transformer depth (4 layers) were chosen based on pilot experiments indicating stable convergence without over-parameterization, and are comparable to configurations commonly used in multimodal biomedical Transformer architectures.

fMRI Patch Embedding with Positional Encoding: The fMRI volumes (48 × 48 × 24 voxels, 80 timepoints) were partitioned into non-overlapping 3D patches of size 6 × 6 × 4 voxels across spatial dimensions, preserving the temporal dimension. Each patch was flattened and projected to a hidden dimension (d_model = 256) via a linear layer, followed by Layer Normalization, GELU activation, and dropout (p = 0.2). Fixed learnable positional embeddings of shape (N_patches, d_model) were added to preserve spatial order and temporal locality.

SNP Embedding with Learned Attention Weights: Each of the 800 SNPs was encoded using a trainable embedding layer mapping genotype values {0, 1, 2} to a d_SNP = 64-dimensional vector. To capture variable importance, an attention mechanism computed scalar weights for each SNP embedding, normalized via a SoftMax function [43]-[46]. The weighted embeddings were flattened and linearly projected into the shared hidden dimension (d_model), forming the SNP modality representation [47]-[49].

Clinical Projection withLayerNorm+ GELU: The 15 clinical features were processed through a fully connected layer mapping to d_model, followed by Layer Normalization, GELU activation, and dropout. This ensured scale normalization and non-linear transformation of heterogeneous clinical measurements.

Modality Tokens and Fusion: Each modality was prepended with a learnable modality token, enabling the Transformer to treat them as global context vectors during self-attention. The resulting token-patched sequences from all three modalities were concatenated along the sequence dimension and passed into a 4-layer Transformer encoder (n_heads = 8, d_ff = 512) with GELU activation.

Classification Head: The output corresponding to the first token in the sequence was passed through a classification head consisting of a linear layer, Layer Normalization, GELU activation, dropout, and a final sigmoid activation to produce the probability of TRD.

The Enhanced Multimodal Transformer (EMT) was trained using the AdamW optimizer, which decouples weight decay from gradient updates to improve generalization in deep neural networks. The learning rate was initialized at 3 × 10⁻⁴ with a weight decay coefficient of 1 × 10⁻⁴. To dynamically adjust learning rates based on validation performance, a ReduceLROnPlateau scheduler was employed, monitoring the validation Area Under the ROC Curve (AUC). If no improvement was observed for three consecutive epochs, the learning rate was reduced by a factor of 0.5. To contextualize the performance of the Enhanced Multimodal Transformer (EMT), two baseline models were implemented: 1) a late-fusion multilayer perceptron (MLP), in which modality-specific embeddings were concatenated and passed through fully connected layers, and 2) a Random Forest classifier trained on concatenated summary features extracted from each modality. Both baselines were optimized using the same training/validation splits and evaluated with identical metrics to ensure fair comparison.

Loss Function: Binary cross-entropy (BCE) loss was used to optimize the binary classification objective [50][51]. Given the predicted probability y ^ and true label y ∈ {0, 1} the loss for a batch of size NNN was computed as:

Early Stopping:To mitigate overfitting and reduce unnecessary computation, early stopping was implemented with a patience of 7 epochs. Training was terminated if the validation AUC did not improve for seven consecutive epochs, and the model parameters from the epoch with the highest AUC were restored for final evaluation.

Evaluation Metrics: Performance was assessed using multiple complementary metrics:

Accuracy: Proportion of correctly classified samples.F1 Score: Harmonic mean of precision and recall, providing a balanced measure for binary classification.Area Under the ROC Curve (AUC): Measures separability of classes across all decision thresholds.ROC Curve: Plots true positive rate versus false positive rate across thresholds.Precision-Recall (PR) Curve: Evaluates model precision against recall, especially relevant for imbalanced classification scenarios.

The model was trained on mini batches of size 12, with shuffling applied to the training set to ensure statistical diversity across epochs. All experiments were conducted on a single CPU for reproducibility of synthetic data experiments, although the architecture is GPU-compatible for large-scale applications.

The Enhanced Multimodal Transformer (EMT) exhibited rapid convergence on the synthetic TRD prediction task. As shown in Figure 1Figure 1, both training and validation losses decreased sharply within the first three epochs, with validation loss stabilizing near zero. Concurrently, the validation AUC rose to 1.00 by epoch two and remained at this ceiling for the remainder of training. Early stopping was triggered at epoch eight, ensuring that the final model preserved the optimal validation AUC without overfitting. The close alignment between training and validation curves indicates strong generalization within the synthetic distribution, reflecting the model’s ability to learn the designed cross-modal patterns effectively.

Figure 1.Training and validation loss curves with validation AUC overlay.

The ROC curve (Figure 2(a)) demonstrates perfect class separability, with an AUC of 1.00, signifying that the EMT ranked all TRD samples higher than non-TRD samples in probability space. Similarly, the Precision–Recall curve (Figure 2(b)) achieved an average precision score of 1.00, maintaining maximal precision across all recall levels. These results confirm that the model achieved complete probabilistic separation between classes, a performance rarely attainable with real-world biomedical datasets. Both baseline models demonstrated strong performance on the synthetic dataset, achieving AUC values above 0.90; however, neither matched the EMT’s perfect class separability (AUC = 1.00, AP = 1.00). In particular, the late-fusion MLP showed reduced sensitivity to cross-modal interactions, while the Random Forest exhibited limited capacity to model high-dimensional fMRI structure. These comparisons highlight the advantage of Transformer-based cross-modal attention for multimodal psychiatric prediction.

(a) (b)

Figure 2.(a) ROC curves; (b) PR curves.

While probabilistic separability was ideal, fixed-threshold classification revealed a notable limitation. As shown in Figure 3Figure 3, predicted probabilities for non-TRD cases clustered tightly around ≈ 0.32, while TRD cases clustered around ≈ 0.41. The absence of probabilities near the conventional 0.5 decision threshold led to systematic misclassification of all TRD cases, resulting in an overall test accuracy of 50% despite perfect AUC and PR performance. This phenomenon reflects a calibration misalignment, in which relative ranking is correct, but absolute probability magnitudes are compressed into a narrow, overlapping range.

Figure 3.Density plots of predicted probabilities per class.

Feature analysis (Figure 4Figure 4) revealed that clinical severity, illness duration, and episode count were the most influential clinical predictors of TRD status. In the genomic modality, SNPs within the first 100 positions engineered as predictive loci contributed disproportionately to classification, consistent with their altered allele frequency distributions between TRD and non-TRD groups. The fMRI modality highlighted activation contrasts in the amygdala and prefrontal cortex, with TRD cases showing hyperactivation in the amygdala and hypoactivation in the prefrontal cortex. These findings confirm that the EMT effectively learned the intended multimodal interaction patterns embedded in the synthetic dataset.

Figure 4.Aggregated feature importance across modalities.

Representative synthetic fMRI activation maps (Figure 5Figure 5) display clear regional contrasts: TRD cases exhibit elevated signal intensity in the amygdala alongside diminished prefrontal activity, while non-TRD cases show stronger anterior cingulate activation. SNP allele distribution plots (Figure 6Figure 6) further confirm class-specific enrichment patterns, with TRD cases showing higher frequencies of heterozygous and homozygous alternate genotypes in the predictive SNP set.

Confidence distribution plots (Figure 7Figure 7) illustrate the probability compression effect: despite perfect separation in rank ordering, both classes occupy narrowly defined probability intervals. To address probability compression, we conducted a post-hoc calibration analysis using Platt scaling and isotonic regression applied to validation-set predictions. Both methods preserved perfect rank ordering (AUC = 1.00) while expanding the probability range toward clinically meaningful values. After calibration, 0.5-threshold accuracy improved substantially, demonstrating that the observed performance limitation stemmed from calibration rather than discriminative failure. This outcome underscores the importance of post-hoc calibration techniques such as Platt scaling or isotonic regression before clinical deployment, to align decision thresholds with true probability estimates. Without calibration, a model can demonstrate flawless AUC yet fail in binary decision-making, highlighting the non-equivalence of ranking performance and thresholder classification accuracy [52]-[54].

Figure 5.Example fMRI slices with annotated regions.

Figure 6.SNP allele frequency histograms for TRD vs non-TRD.

Figure 7.Violin plots or boxplots showing confidence distributions for each class.