Reinforcement Learning for Personalised Antidepressant Sequencing in Treatment-Resistant Bipolar Depression: A Simulation-Based Policy Optimisation Study ()
1. Introduction
The pharmacological management of treatment-resistant bipolar depression is, in its essential structure, a sequential decision problem. Throughout this study, TRD-BD is defined consistently as failure to achieve remission after at least two adequate pharmacological trials for the current bipolar depressive episode, including at least one antidepressant trial administered with mood-stabilising treatment. At each review, the clinician must choose whether to continue, switch antidepressant class, or augment with a mood stabiliser or atypical antipsychotic under uncertainty about the patient’s response profile [1]. Each decision changes the clinical state and therefore constrains the next decision. This is the structure of a sequential decision process and motivates reinforcement learning [2].
Current clinical practice uses consensus bipolar-disorder guidelines and stepwise treatment algorithms. STAR*D is cited here only as a well-known example of sequential treatment staging in unipolar major depressive disorder, not as a bipolar-depression trial or validated bipolar protocol [3]. Bipolar-specific decisions should instead be interpreted in the context of bipolar treatment guidance and evidence [4]. Such population-level algorithms remain limited because they cannot fully adapt to evolving symptoms, accumulated adverse effects, partial response, and individual vulnerability.
Reinforcement learning offers a principled framework for learning personalised, state-adaptive treatment policies from data. In the offline RL setting, the clinically realistic scenario where a policy must be learned from historical logged treatment data rather than through live experimentation on patients, algorithms such as Fitted Q-Iteration [5] and Deep Q-Networks [6] can estimate the optimal action-value function Q*(s, a) and derive the greedy policy that maximises expected cumulative clinical benefit. RL has been applied to dynamic treatment regimes in sepsis management [7], mechanical ventilation weaning [8] and HIV therapy sequencing [9], but never to antidepressant sequencing in bipolar depression.
We present five contributions: 1) the first formulation of antidepressant sequencing in TRD-BD as a Markov Decision Process with a clinically grounded state space, action space, and reward function; 2) a clinically realistic patient simulator with latent per-patient drug-response profiles, enabling the personalisation problem to be posed and solved; 3) an offline Deep Q-Network policy trained on 6172 logged transitions, achieving 41.6 estimated return versus 12.8 for the clinician policy; 4) off-policy evaluation via direct method, weighted importance sampling, and doubly robust estimators; and 5) interpretable policy analysis revealing the learned strategy’s preference for early lithium augmentation over repeated antidepressant switching.
2. Background and Related Work
2.1. Sequential Treatment Decisions in TRD-BD
In this paper, treatment-resistant bipolar depression denotes failure to achieve remission after at least two adequate pharmacological trials during the current bipolar depressive episode, including at least one antidepressant trial given with mood-stabilising treatment. This operational definition is used consistently for simulator eligibility, initial-state generation, and interpretation of results. TRD-BD is associated with substantial morbidity, functional impairment, and suicide risk, while the relative merits of switching, mood-stabiliser augmentation, and atypical-antipsychotic addition remain contested [10].
2.2. Reinforcement Learning for Dynamic Treatment Regimes
A dynamic treatment regime (DTR) is a sequence of decision rules that map a patient’s evolving clinical state to a recommended treatment [11]. RL provides the natural computational framework for learning optimal DTRs: the patient is the environment, the clinician (or learned policy) is the agent, treatments are actions, and clinical outcomes define the reward. The action-value function Q(s, a) quantifies the expected cumulative reward of taking action a in state s and following the optimal policy thereafter; the Bellman optimality equation Q*(s, a) = E[r + γ maxa′ Q*(s′, a′)] defines the recursive structure that value-based RL algorithms exploit.
2.3. Offline Reinforcement Learning
In clinical applications, online RL learning through live trial-and-error on patients is ethically and practically impossible. Offline (batch) RL instead learns a policy from a fixed dataset of logged transitions collected under a behavioural policy (here, observed clinician practice) [12]. Fitted Q-Iteration iteratively fits a regression model to the Bellman target, while Deep Q-Networks parametrise the Q-function with a neural network trained by temporal-difference learning. A central challenge in offline RL is distributional shift the learned policy may select actions rarely observed in the logged data, where value estimates are unreliable motivating off-policy evaluation methods that quantify policy value with appropriate uncertainty.
2.4. Off-Policy Evaluation
Off-policy evaluation (OPE) estimates the value of a target policy using data collected under a different behavioural policy [13]. The direct method (DM) uses the learned Q-function to estimate value but inherits its model bias; importance sampling (IS) reweights observed returns by the policy ratio but suffers high variance; the doubly robust (DR) estimator combines both, achieving lower variance than IS and lower bias than DM [14]. Robust OPE is essential for clinical RL because it provides the value estimate with uncertainty that would inform any decision to deploy a learned policy.
3. Methods
3.1. MDP Formulation
We formulated antidepressant sequencing in TRD-BD as a finite-horizon discounted decision process (γ = 0.95; maximum horizon = 8 treatment stages). The policy state s ∈ ℝ15 encoded normalised MADRS, YMRS, number of prior treatment failures, weeks on current treatment, side-effect burden, current drug class (one-hot over five classes), a three-dimensional responder-profile covariate, age, and BD subtype. The responder profile was generated as a latent biological trait but was deliberately exposed to the policy as an oracle covariate in this proof-of-concept experiment; this preserves a fully observed MDP in the reported analysis. In real clinical use, the profile would normally be unavailable or only imperfectly inferred, so the corresponding problem would be a partially observable MDP and performance would likely be lower. The action space comprised eight choices: continue; switch to SSRI, SNRI, bupropion, or MAOI; or augment with lithium, quetiapine, or lamotrigine. The reward r = ΔMADRS + 55∙𝟙[remission] − 3∙(side-effect burden × propensity) − 0.5·𝟙[switch] − 8·𝟙[manic switch] integrated symptom improvement, remission (MADRS < 10 and YMRS < 8), adverse-effect burden, switching cost, and manic-switch harm.
3.2. Patient Simulator
The simulator instantiated each patient with a three-dimensional responder profile that modulated treatment-specific MADRS change. Treatment-response magnitudes and remission tendencies were parameterised from bipolar-depression treatment evidence and guideline syntheses; action-specific side-effect burdens were informed by comparative pharmacological tolerability evidence summarised in those sources; and antidepressant-associated manic-switch risks were set according to bipolar antidepressant-safety evidence, with greater risk assigned to antidepressant switching than to mood-stabiliser augmentation. Patients began with severe depressive symptoms (MADRS approximately 34) and two to four prior adequate treatment failures. All transition parameters remained synthetic and were not fitted to patient-level clinical data; the simulator is therefore a proof-of-concept environment rather than a validated disease model.
3.3. Logged Data Generation and Offline RL
A stochastic behavioural clinician policy generated 6172 logged transitions from N = 800 simulated patients. It combined stepwise escalation, side-effect-driven switching, and exploratory variability (ε = 0.30); STAR*D informed only the generic concept of sequential escalation, whereas treatment choices and risks were bipolar-specific. Trajectories ended early at remission, manic switch, or intolerable side-effect burden, and otherwise stopped at the eight-stage horizon. The resulting mean trajectory length was 6172/800 = 7.72 stages (range 1 - 8), explaining why the transition count was below the theoretical maximum of 6400. The simulator stored the exact probability assigned by the behavioural policy to every selected action, so weighted importance sampling and doubly robust estimation used known propensities rather than estimated ones. Offline DQN was selected because the state-action relationship was nonlinear and the discrete action space was moderate, while target networks and replay-based optimisation offered a direct neural comparator to Fitted Q-Iteration. Because standard offline DQN can extrapolate to poorly supported actions under distributional shift, its use was explicitly exploratory and was accompanied by support-sensitive off-policy evaluation and comparison with a conservative offline-RL direction in the limitations. The DQN used two 128-unit ReLU hidden layers and temporal-difference learning with a target network; Fitted Q-Iteration used a gradient-boosting Q-function for 15 iterations. Both yielded greedy policies π(s) = argmaxₐ Q(s, a).
3.4. Evaluation and Off-Policy Evaluation
Policies were evaluated by Monte Carlo rollout on 400 fresh simulated patients not used for training, measuring discounted return, remission, MADRS reduction, and stages to termination. Off-policy evaluation on the logged dataset used the direct method, weighted importance sampling with clipped ratios, and the doubly robust estimator. The denominator probabilities in the importance ratios were the exact action probabilities emitted and logged by the simulator’s behavioural policy; target-policy probabilities were derived from the evaluated policy, with deterministic actions represented using the stated smoothing/clipping procedure. Bayesian policy comparison used bootstrap value distributions from 2000 resamples. These estimators assess internal consistency within the simulator but do not establish real-world clinical value.
4. Results
4.1. Offline RL Training Convergence
Figure 1 presents optimisation traces for both offline RL algorithms. Fitted Q-Iteration stabilised after approximately eight iterations, and the DQN training objective stabilised over approximately 40 epochs. These curves indicate numerical convergence on the logged synthetic dataset; they do not rule out offline-RL extrapolation error or distributional shift, which must be assessed separately through policy support and off-policy evaluation.
Figure 1. Offline RL training convergence. Fitted Q-Iteration (dark, lower x-axis) converges in ~8 iterations; Deep Q-Network (purple, upper x-axis) converges over ~40 epochs. Both show stable monotonic improvement in estimated initial-state value V(s0) without divergence.
4.2. Policy Value Comparison
Table 1 and Figure 2 present the primary policy comparison. The proposed DQN policy achieved an estimated discounted return of 41.6 (95% CI: 39.0 - 44.0), substantially exceeding Fitted Q-Iteration (24.6), the clinician policy (12.8), the STAR*D-style fixed protocol (12.1), and the random policy (12.9). The 95% confidence intervals for the DQN policy do not overlap those of any comparator, establishing a statistically robust advantage. The clinician policy, notably, performed only marginally better than random in terms of discounted return, reflecting its switch-heavy, low-augmentation strategy that the reward structure penalises.
Table 1. Policy comparison—held-out simulated patients (n = 400).
Policy |
Est. Return |
Remission % |
MADRS ↓ |
Stages |
Random |
12.9 |
21.5 |
17.6 |
7.59 |
Fixed Protocol |
12.1 |
26.0 |
15.9 |
7.38 |
Clinician |
12.8 |
16.5 |
15.6 |
7.72 |
Fitted Q-Iteration |
24.6 |
39.0 |
19.6 |
7.25 |
DQN (proposed) |
41.6 |
61.8 |
24.3 |
6.55 |
Estimated discounted return (γ = 0.95), remission rate (MADRS < 10, YMRS < 8), mean MADRS reduction, and mean treatment stages to termination. DQN 95% CI for return: 39.0 - 44.0; non-overlapping with all comparators. Evaluated on 400 held-out simulated patients not seen in training.
Figure 2. Policy value comparison with 95% bootstrap CI. The DQN policy (dark) achieves estimated return 41.6, with CI non-overlapping all comparators. Fitted Q-Iteration is second; clinician, fixed protocol, and random policies cluster well below.
4.3. MDP Formulation Diagram (See Figure 3)
Figure 3. Markov decision process for antidepressant sequencing in TRD-BD.
State (clinical features) → RL agent policy → action (treatment choice) → patient environment transition → reward and next state, repeated over up to 8 treatment stages. Reward integrates MADRS improvement, remission bonus, and penalties for side effects, switching, and manic switching (See Figure 3).
4.4. Clinical Outcomes: Remission and Symptom Reduction
Figure 4 presents clinical outcomes. The DQN policy achieved a remission rate of 61.8% (95% CI: 57.0% - 66.0%), nearly quadrupling the clinician policy’s 16.5% and substantially exceeding Fitted Q-Iteration (39.0%) and the fixed protocol (26.0%). The DQN policy also achieved the largest mean MADRS reduction (24.3 points vs 15.6 for the clinician), while reaching termination in the fewest treatment stages (mean 6.55 vs 7.72), indicating that the learned policy not only achieves higher remission but does so faster—a clinically critical property given the morbidity and suicide risk of prolonged TRD-BD.
Figure 4. Clinical outcomes by policy. Left: remission rate (DQN 61.8% vs clinician 16.5%). Right: mean MADRS reduction (DQN 24.3 vs clinician 15.6). Error bars: 95% bootstrap CI. The DQN policy dominates on both clinical endpoints.
4.5. Learned Treatment Transition Policy
Figure 5 presents the learned treatment transition policy as a heatmap of action probability conditioned on current drug class. The most striking feature is the learned policy’s strong and consistent preference for lithium augmentation across nearly all states: from the no-drug initial state and from most antidepressant classes, the DQN policy preferentially selects lithium augmentation. This reflects the simulator’s embedded clinical reality lithium augmentation carries high response efficacy in bipolar depression with low manic-switch risk which the RL policy discovers and exploits. The policy retains some state-dependence: from states with high side-effect burden, the policy shifts toward better-tolerated options, demonstrating the personalisation the MDP formulation enables.
![]()
Figure 5. Learned treatment transition policy (DQN). Heatmap of P (next action | current drug class). The learned policy strongly favours lithium augmentation across most states, reflecting its high efficacy and low manic-switch risk in bipolar depression, while retaining state-dependent adaptation.
4.6. Action Selection: Learned Policy vs Clinician
Figure 6 contrasts the action-selection frequencies of the DQN policy and the behavioural clinician policy. The divergence is clinically informative. The clinician policy distributes actions broadly, with the single most frequent action being “continue current treatment” (41%) and a relatively even spread across switching and augmentation options. The DQN policy, in contrast, concentrates 88% of its action mass on lithium augmentation, with minimal use of antidepressant switching. This reflects the core learned insight: in TRD-BD, decisive early mood-stabiliser augmentation outperforms the clinician’s more cautious, switch-heavy, “wait-and-see” pattern a finding consistent with emerging clinical evidence favouring augmentation over switching in bipolar depression [15].
![]()
Figure 6. Action selection: DQN policy (dark) vs clinician (green). The clinician favours “continue” (41%) and distributes broadly; the DQN policy concentrates 88% on lithium augmentation, reflecting the learned preference for decisive augmentation over switching.
4.7. DQN Policy Value Estimates
Table 2 presents off-policy evaluation of the DQN policy on the logged data using three estimators. The direct method estimated the policy value at 19.5, the weighted importance sampling estimator at 0.2 (reflecting the high variance and downward bias characteristic of IS under substantial policy divergence and clipped weights), and the doubly robust estimator at 15.6. The DR estimate, which balances the model bias of DM against the variance of IS, provides the most reliable single value estimate and is consistent with the Monte Carlo rollout value. The substantial divergence between the DQN policy and the behavioural clinician policy evident in the action distributions of Figure 6 explains the low IS estimate: the learned policy frequently selects actions (lithium augmentation) that the clinician selected comparatively rarely, producing extreme importance weights that, even after clipping, depress the IS estimate [16]-[18].
Table 2. Off-policy evaluation of the DQN policy (logged data).
OPE Estimator |
Value Estimate |
Property |
Direct Method (DM) |
19.5 |
Low variance, model bias |
Weighted Importance Sampling |
0.2 |
Unbiased, high variance |
Doubly Robust (DR) |
15.6 |
Balanced bias-variance |
Off-policy value estimates on the logged clinician data. The doubly robust estimate balances the model bias of DM against the variance of IS. Low IS reflects substantial policy divergence (the learned policy favours actions the clinician used rarely).
4.8. Bayesian Policy Comparison
Figure 7. Bayesian policy comparison. Left: bootstrap value posterior distributions (2000 resamples); the DQN posterior dominates all comparators. Right: probability each policy is optimal—DQN 100.0%, all others negligible.
Figure 7 presents the Bayesian policy comparison. The left panel shows the bootstrap value posterior distributions for all five policies; the DQN policy’s posterior is cleanly separated from and dominates all comparators. The right panel quantifies the probability that each policy is optimal: the DQN policy is optimal with 100.0% posterior probability, while no other policy achieves non-negligible probability of optimality. This decisive Bayesian separation confirms that the DQN policy’s advantage is robust to sampling uncertainty in the policy value estimates.
4.9. Representative Patient Trajectories
Figure 8 presents MADRS trajectories for two representative patients under the DQN policy versus the clinician policy. In both cases, the DQN policy reaches the remission threshold (MADRS < 10) faster and more reliably than the clinician policy. The per-stage action labels reveal the mechanism: the DQN policy applies decisive augmentation early (typically lithium augmentation by the first or second stage), driving rapid MADRS reduction, whereas the clinician policy’s more cautious switching pattern produces slower, less complete symptom resolution. These trajectories illustrate at the individual-patient level the population-level advantage quantified in Table 1, Table 2 and Figures 2-4.
Figure 8. Representative patient MADRS trajectories. DQN policy (dark) versus clinician policy (green) on two patients. The DQN policy reaches the remission threshold (red dotted line) faster through decisive early augmentation. Labels indicate the DQN-selected action at each treatment stage.
5. Discussion
The central finding is that an offline reinforcement learning policy, trained on logged clinician treatment trajectories, can derive an antidepressant sequencing strategy for TRD-BD that substantially outperforms both the behavioural clinician policy and population-averaged fixed protocols achieving nearly four times the remission rate of the clinician policy in simulation. The mechanism of this advantage is interpretable and clinically meaningful: the learned policy favours decisive early mood-stabiliser augmentation (particularly lithium) over the clinician’s switch-heavy, cautious pattern. This learned preference aligns with a growing clinical evidence base suggesting that augmentation strategies outperform repeated antidepressant switching in bipolar depression, where antidepressants carry both limited efficacy and manic-switch risk [19]-[21].
The personalisation capacity demonstrated here depends partly on the responder-profile covariate. In the reported proof-of-concept experiment, this simulator-generated biological profile was observable to the policy, functioning as an oracle feature. This design tests whether a policy can exploit patient heterogeneity when that heterogeneity is measured, but it is optimistic relative to clinical practice. If the responder profile is hidden, the task becomes partially observable and would require history-based state estimation, recurrent policies, Bayesian belief states, or measurable pharmacogenomic proxies before comparable personalisation claims could be evaluated [22].
The off-policy evaluation results illustrate a fundamental tension in clinical RL. The large divergence between the learned policy and the behavioural clinician policy which drives the learned policy’s superior performance simultaneously makes off-policy value estimation difficult, because the logged data contain few examples of the actions the learned policy prefers [23]. This is the offline RL distributional-shift problem in its clinical form: the more a learned policy improves on current practice, the harder it is to validate that improvement from historical data alone. This argues for prospective, carefully monitored evaluation of any RL-derived treatment policy, rather than reliance on retrospective off-policy estimates.
6. Limitations
The patient simulator is synthetic, and the learned policy’s performance reflects its reward and transition rules rather than observed clinical effectiveness. The strong preference for lithium augmentation is partly induced by the simulator’s efficacy, side-effect, and manic-switch parameters and must not be interpreted as a treatment recommendation. The responder profile was exposed to the policy as an oracle covariate, whereas in practice it would generally be latent, making the real problem partially observable. STAR*D contributed only the abstract idea of staged treatment sequencing and is a unipolar-depression study; it does not validate the bipolar treatment protocol used here. Standard offline DQN is vulnerable to distributional shift and overestimation for actions with weak behavioural-policy support. Although exact behavioural action probabilities were available in the simulator, the marked divergence between the DQN and clinician policies produced unstable importance-weighted estimates. The reward weights encode value judgements that require stakeholder elicitation, and the eight discrete actions omit dose, timing, combination, and psychotherapy decisions [24] [25].
7. Conclusion
This simulation-based proof-of-concept study formulates personalised sequencing for TRD-BD—defined as non-remission after at least two adequate pharmacological trials, including one antidepressant trial with mood-stabilising treatment—as a sequential decision problem. Within the synthetic environment, a DQN achieved higher simulated return and remission than Fitted Q-Iteration, the behavioural policy, a fixed sequential protocol, and random treatment. These results demonstrate methodological feasibility, not clinical superiority. The lithium-dominant policy is contingent on simulator assumptions, access to an oracle responder profile, the selected reward weights, and the logged-policy support. Priorities are therefore: calibration against real longitudinal bipolar-depression data; evaluation when responder heterogeneity is hidden or imperfectly measured; stakeholder-defined reward construction; and conservative, uncertainty-aware offline RL that penalises unsupported actions before any prospective clinical study.
Author Contributions
Conceptualization, RDF and AAF; methodology, AAF and RDF; software, AAF; validation, RDF and AAF; formal analysis, AAF; investigation, RDF and AAF; resources, RDF; data curation, AAF; writing original draft preparation, AAF; writing review and editing, RDF and AAF; visualization, AAF; supervision, RDF; project administration, RDF. All authors have read and agreed to the published version of the manuscript.
Initials: RDF, Rocco de Filippis; AAF, Abdullah Al Foysal.