In this paper, treatment-resistant bipolar depression denotes failure to achieve remission after at least two adequate pharmacological trials during the current bipolar depressive episode, including at least one antidepressant trial given with mood-stabilising treatment. This operational definition is used consistently for simulator eligibility, initial-state generation, and interpretation of results. TRD-BD is associated with substantial morbidity, functional impairment, and suicide risk, while the relative merits of switching, mood-stabiliser augmentation, and atypical-antipsychotic addition remain contested [10].

A dynamic treatment regime (DTR) is a sequence of decision rules that map a patient’s evolving clinical state to a recommended treatment [11]. RL provides the natural computational framework for learning optimal DTRs: the patient is the environment, the clinician (or learned policy) is the agent, treatments are actions, and clinical outcomes define the reward. The action-value function Q(s, a) quantifies the expected cumulative reward of taking action a in state s and following the optimal policy thereafter; the Bellman optimality equation Q*(s, a) = E[r + γ max_a′ Q*(s′, a′)] defines the recursive structure that value-based RL algorithms exploit.

In clinical applications, online RL learning through live trial-and-error on patients is ethically and practically impossible. Offline (batch) RL instead learns a policy from a fixed dataset of logged transitions collected under a behavioural policy (here, observed clinician practice) [12]. Fitted Q-Iteration iteratively fits a regression model to the Bellman target, while Deep Q-Networks parametrise the Q-function with a neural network trained by temporal-difference learning. A central challenge in offline RL is distributional shift the learned policy may select actions rarely observed in the logged data, where value estimates are unreliable motivating off-policy evaluation methods that quantify policy value with appropriate uncertainty.

Off-policy evaluation (OPE) estimates the value of a target policy using data collected under a different behavioural policy [13]. The direct method (DM) uses the learned Q-function to estimate value but inherits its model bias; importance sampling (IS) reweights observed returns by the policy ratio but suffers high variance; the doubly robust (DR) estimator combines both, achieving lower variance than IS and lower bias than DM [14]. Robust OPE is essential for clinical RL because it provides the value estimate with uncertainty that would inform any decision to deploy a learned policy.

We formulated antidepressant sequencing in TRD-BD as a finite-horizon discounted decision process (γ = 0.95; maximum horizon = 8 treatment stages). The policy state s ∈ ℝ¹⁵ encoded normalised MADRS, YMRS, number of prior treatment failures, weeks on current treatment, side-effect burden, current drug class (one-hot over five classes), a three-dimensional responder-profile covariate, age, and BD subtype. The responder profile was generated as a latent biological trait but was deliberately exposed to the policy as an oracle covariate in this proof-of-concept experiment; this preserves a fully observed MDP in the reported analysis. In real clinical use, the profile would normally be unavailable or only imperfectly inferred, so the corresponding problem would be a partially observable MDP and performance would likely be lower. The action space comprised eight choices: continue; switch to SSRI, SNRI, bupropion, or MAOI; or augment with lithium, quetiapine, or lamotrigine. The reward r = ΔMADRS + 55∙𝟙[remission] − 3∙(side-effect burden × propensity) − 0.5·𝟙[switch] − 8·𝟙[manic switch] integrated symptom improvement, remission (MADRS < 10 and YMRS < 8), adverse-effect burden, switching cost, and manic-switch harm.

The simulator instantiated each patient with a three-dimensional responder profile that modulated treatment-specific MADRS change. Treatment-response magnitudes and remission tendencies were parameterised from bipolar-depression treatment evidence and guideline syntheses; action-specific side-effect burdens were informed by comparative pharmacological tolerability evidence summarised in those sources; and antidepressant-associated manic-switch risks were set according to bipolar antidepressant-safety evidence, with greater risk assigned to antidepressant switching than to mood-stabiliser augmentation. Patients began with severe depressive symptoms (MADRS approximately 34) and two to four prior adequate treatment failures. All transition parameters remained synthetic and were not fitted to patient-level clinical data; the simulator is therefore a proof-of-concept environment rather than a validated disease model.

A stochastic behavioural clinician policy generated 6172 logged transitions from N = 800 simulated patients. It combined stepwise escalation, side-effect-driven switching, and exploratory variability (ε = 0.30); STAR*D informed only the generic concept of sequential escalation, whereas treatment choices and risks were bipolar-specific. Trajectories ended early at remission, manic switch, or intolerable side-effect burden, and otherwise stopped at the eight-stage horizon. The resulting mean trajectory length was 6172/800 = 7.72 stages (range 1 - 8), explaining why the transition count was below the theoretical maximum of 6400. The simulator stored the exact probability assigned by the behavioural policy to every selected action, so weighted importance sampling and doubly robust estimation used known propensities rather than estimated ones. Offline DQN was selected because the state-action relationship was nonlinear and the discrete action space was moderate, while target networks and replay-based optimisation offered a direct neural comparator to Fitted Q-Iteration. Because standard offline DQN can extrapolate to poorly supported actions under distributional shift, its use was explicitly exploratory and was accompanied by support-sensitive off-policy evaluation and comparison with a conservative offline-RL direction in the limitations. The DQN used two 128-unit ReLU hidden layers and temporal-difference learning with a target network; Fitted Q-Iteration used a gradient-boosting Q-function for 15 iterations. Both yielded greedy policies π(s) = argmaxₐ Q(s, a).

Policies were evaluated by Monte Carlo rollout on 400 fresh simulated patients not used for training, measuring discounted return, remission, MADRS reduction, and stages to termination. Off-policy evaluation on the logged dataset used the direct method, weighted importance sampling with clipped ratios, and the doubly robust estimator. The denominator probabilities in the importance ratios were the exact action probabilities emitted and logged by the simulator’s behavioural policy; target-policy probabilities were derived from the evaluated policy, with deterministic actions represented using the stated smoothing/clipping procedure. Bayesian policy comparison used bootstrap value distributions from 2000 resamples. These estimators assess internal consistency within the simulator but do not establish real-world clinical value.

Figure 1Figure 1 presents optimisation traces for both offline RL algorithms. Fitted Q-Iteration stabilised after approximately eight iterations, and the DQN training objective stabilised over approximately 40 epochs. These curves indicate numerical convergence on the logged synthetic dataset; they do not rule out offline-RL extrapolation error or distributional shift, which must be assessed separately through policy support and off-policy evaluation.

Figure 1. Offline RL training convergence. Fitted Q-Iteration (dark, lower x-axis) converges in ~8 iterations; Deep Q-Network (purple, upper x-axis) converges over ~40 epochs. Both show stable monotonic improvement in estimated initial-state value V(s₀) without divergence.

Table1 and Figure 2 present the primary policy comparison. The proposed DQN policy achieved an estimated discounted return of 41.6 (95% CI: 39.0 - 44.0), substantially exceeding Fitted Q-Iteration (24.6), the clinician policy (12.8), the STAR*D-style fixed protocol (12.1), and the random policy (12.9). The 95% confidence intervals for the DQN policy do not overlap those of any comparator, establishing a statistically robust advantage. The clinician policy, notably, performed only marginally better than random in terms of discounted return, reflecting its switch-heavy, low-augmentation strategy that the reward structure penalises.

Table 1.Policy comparison—held-out simulated patients (n = 400).

Estimated discounted return (γ = 0.95), remission rate (MADRS < 10, YMRS < 8), mean MADRS reduction, and mean treatment stages to termination. DQN 95% CI for return: 39.0 - 44.0; non-overlapping with all comparators. Evaluated on 400 held-out simulated patients not seen in training.

Figure 2. Policy value comparison with 95% bootstrap CI. The DQN policy (dark) achieves estimated return 41.6, with CI non-overlapping all comparators. Fitted Q-Iteration is second; clinician, fixed protocol, and random policies cluster well below.

Figure 3. Markov decision process for antidepressant sequencing in TRD-BD.

State (clinical features) → RL agent policy → action (treatment choice) → patient environment transition → reward and next state, repeated over up to 8 treatment stages. Reward integrates MADRS improvement, remission bonus, and penalties for side effects, switching, and manic switching (See Figure 3).

Figure 4Figure 4 presents clinical outcomes. The DQN policy achieved a remission rate of 61.8% (95% CI: 57.0% - 66.0%), nearly quadrupling the clinician policy’s 16.5% and substantially exceeding Fitted Q-Iteration (39.0%) and the fixed protocol (26.0%). The DQN policy also achieved the largest mean MADRS reduction (24.3 points vs 15.6 for the clinician), while reaching termination in the fewest treatment stages (mean 6.55 vs 7.72), indicating that the learned policy not only achieves higher remission but does so faster—a clinically critical property given the morbidity and suicide risk of prolonged TRD-BD.

Figure 4. Clinical outcomes by policy. Left: remission rate (DQN 61.8% vs clinician 16.5%). Right: mean MADRS reduction (DQN 24.3 vs clinician 15.6). Error bars: 95% bootstrap CI. The DQN policy dominates on both clinical endpoints.

Figure 5Figure 5 presents the learned treatment transition policy as a heatmap of action probability conditioned on current drug class. The most striking feature is the learned policy’s strong and consistent preference for lithium augmentation across nearly all states: from the no-drug initial state and from most antidepressant classes, the DQN policy preferentially selects lithium augmentation. This reflects the simulator’s embedded clinical reality lithium augmentation carries high response efficacy in bipolar depression with low manic-switch risk which the RL policy discovers and exploits. The policy retains some state-dependence: from states with high side-effect burden, the policy shifts toward better-tolerated options, demonstrating the personalisation the MDP formulation enables.

Figure 5. Learned treatment transition policy (DQN). Heatmap of P (next action | current drug class). The learned policy strongly favours lithium augmentation across most states, reflecting its high efficacy and low manic-switch risk in bipolar depression, while retaining state-dependent adaptation.

Figure 6Figure 6 contrasts the action-selection frequencies of the DQN policy and the behavioural clinician policy. The divergence is clinically informative. The clinician policy distributes actions broadly, with the single most frequent action being “continue current treatment” (41%) and a relatively even spread across switching and augmentation options. The DQN policy, in contrast, concentrates 88% of its action mass on lithium augmentation, with minimal use of antidepressant switching. This reflects the core learned insight: in TRD-BD, decisive early mood-stabiliser augmentation outperforms the clinician’s more cautious, switch-heavy, “wait-and-see” pattern a finding consistent with emerging clinical evidence favouring augmentation over switching in bipolar depression [15].

Figure 6. Action selection: DQN policy (dark) vs clinician (green). The clinician favours “continue” (41%) and distributes broadly; the DQN policy concentrates 88% on lithium augmentation, reflecting the learned preference for decisive augmentation over switching.

Table2 presents off-policy evaluation of the DQN policy on the logged data using three estimators. The direct method estimated the policy value at 19.5, the weighted importance sampling estimator at 0.2 (reflecting the high variance and downward bias characteristic of IS under substantial policy divergence and clipped weights), and the doubly robust estimator at 15.6. The DR estimate, which balances the model bias of DM against the variance of IS, provides the most reliable single value estimate and is consistent with the Monte Carlo rollout value. The substantial divergence between the DQN policy and the behavioural clinician policy evident in the action distributions of Figure 6Figure 6 explains the low IS estimate: the learned policy frequently selects actions (lithium augmentation) that the clinician selected comparatively rarely, producing extreme importance weights that, even after clipping, depress the IS estimate [16]-[18].

Table 2.Off-policy evaluation of the DQN policy (logged data).

Off-policy value estimates on the logged clinician data. The doubly robust estimate balances the model bias of DM against the variance of IS. Low IS reflects substantial policy divergence (the learned policy favours actions the clinician used rarely).

Figure 7. Bayesian policy comparison. Left: bootstrap value posterior distributions (2000 resamples); the DQN posterior dominates all comparators. Right: probability each policy is optimal—DQN 100.0%, all others negligible.

Figure 7Figure 7 presents the Bayesian policy comparison. The left panel shows the bootstrap value posterior distributions for all five policies; the DQN policy’s posterior is cleanly separated from and dominates all comparators. The right panel quantifies the probability that each policy is optimal: the DQN policy is optimal with 100.0% posterior probability, while no other policy achieves non-negligible probability of optimality. This decisive Bayesian separation confirms that the DQN policy’s advantage is robust to sampling uncertainty in the policy value estimates.

Figure 8Figure 8 presents MADRS trajectories for two representative patients under the DQN policy versus the clinician policy. In both cases, the DQN policy reaches the remission threshold (MADRS < 10) faster and more reliably than the clinician policy. The per-stage action labels reveal the mechanism: the DQN policy applies decisive augmentation early (typically lithium augmentation by the first or second stage), driving rapid MADRS reduction, whereas the clinician policy’s more cautious switching pattern produces slower, less complete symptom resolution. These trajectories illustrate at the individual-patient level the population-level advantage quantified in Table1, Table 2 and Figures 2-4.

Figure 8. Representative patient MADRS trajectories. DQN policy (dark) versus clinician policy (green) on two patients. The DQN policy reaches the remission threshold (red dotted line) faster through decisive early augmentation. Labels indicate the DQN-selected action at each treatment stage.