Reinforcement Learning-Based Personalized Depression Treatment Using Synthetic Data and Real-Time Decision Support ()
1. Introduction
Depression is one of the most prevalent and debilitating mental health disorders globally, significantly affecting quality of life, social functioning, and economic productivity [1]-[3]. Despite the availability of multiple antidepressant medications, treatment selection remains largely empirical, relying heavily on a clinician’s judgment and patient-reported outcomes [4]-[7]. The highly individualized nature of depression, coupled with heterogeneous responses to medications, often results in prolonged trial-and-error prescribing [8]-[11]. This conventional approach not only delays optimal treatment but also increases the risk of side effects, patient dissatisfaction, and treatment non-adherence [12]-[16]. Reinforcement Learning (RL) has emerged as a promising solution for sequential decision-making in complex environments, including healthcare [17] [18]. By modelling treatment as a series of dynamic, state-dependent actions, RL frameworks can learn to optimize long-term patient outcomes through continuous interaction with the treatment environment [19]-[22]. Unlike traditional predictive models, RL can adapt to evolving patient states, making it particularly suitable for chronic conditions like depression where symptom patterns and medication responses change over time [23]-[26]. However, the development of effective RL models in psychiatry faces significant challenges, notably the scarcity of large, high-quality datasets and stringent patient privacy regulations [27]-[31]. To address these barriers, this study utilizes synthetically generated patient data, which simulates realistic treatment pathways while preserving confidentiality. The synthetic data includes diverse patient profiles, symptom dynamics, medication effects, and stochastic life events, providing a rich environment for agent training. This research introduces a deep Q-learning-based reinforcement learning agent capable of recommending personalized antidepressant treatments in real-time. The system integrates synthetic patient simulation, feature engineering, and a reinforcement learning pipeline to optimize treatment sequences. By incorporating real-time decision-making and dynamic patient monitoring, the proposed framework aims to accelerate the pathway toward precision psychiatry, reducing dependence on traditional trial-and-error prescribing and offering a scalable, privacy-conscious solution for intelligent mental health care.
2. Methods
2.1. Synthetic Data Generation
To address the challenges of limited clinical data availability and patient privacy constraints, a comprehensive synthetic depression patient simulator was developed [32]-[34]. This simulator generated longitudinal symptom data for 1000 synthetic patients, each tracked over a 180-day period. The simulation incorporated five key depressive symptoms inspired by the PHQ-9 scale: mood, sleep, energy, concentration, and appetite. Each patient’s daily symptom profile was influenced by multiple interacting factors to closely resemble real-world treatment dynamics. Medications were randomly assigned from a predefined list of commonly used antidepressants, including Sertraline, Escitalopram, Fluoxetine, Venlafaxine, and Bupropion. Each medication was associated with specific therapeutic effects and side-effect profiles, modelled to impact symptoms at varying rates and magnitudes. To enhance realism, medication adherence rates were stochastically assigned using beta distributions to capture patient-specific adherence variability. Additionally, random external events, such as stressful life changes, were incorporated to simulate symptom fluctuations that are common in real clinical settings [35]-[37]. The simulator also modelled key clinical decision points, including medication switches and dosage adjustments. Treatment changes were probabilistically determined based on ongoing symptom severity, duration of medication exposure, and patient response trends, mimicking the sequential decision-making process of human clinicians. To assess the realism of the synthetic patient data, the output of the simulator was compared against known epidemiological benchmarks and clinical datasets derived from PHQ-9-based studies. Symptom score distributions, response curves, and medication adherence patterns were verified to align with real-world clinical observations. Furthermore, an expert validation process was conducted wherein two board-certified psychiatrists independently reviewed a stratified sample of 30 synthetic patient trajectories. Their qualitative assessments focused on plausibility, treatment transitions, and symptom progression. Inter-rater reliability was high, with a Cohen’s kappa of 0.82, indicating strong agreement on the clinical credibility of the simulations. These steps provide preliminary evidence that the synthetic data faithfully represent plausible treatment dynamics and symptom variability found in real-world depression management scenarios.
2.2. Feature Engineering
To enable the reinforcement learning agent to effectively capture temporal patterns and treatment responses, an extensive feature engineering process was applied [38] [39]. This included the calculation of 7-day and 30-day rolling averages for each symptom to provide short-term and long-term context. Lag features were generated at 1-day, 3-day, and 7-day intervals to capture symptom history and recent changes. Additionally, interaction terms, such as dose-adherence multipliers and non-linear transformations like the square root of days on medication, were introduced to model the complex interplay between medication exposure and patient behaviour over time.
2.3. Train-Test Split
For model evaluation, the dataset was partitioned at the patient level to ensure that the reinforcement learning agent was tested exclusively on unseen patient trajectories [40] [41]. This approach prevents data leakage and supports realistic generalization performance assessment. By splitting the data this way, the system’s ability to adapt to new patients and dynamically optimize treatment plans in a generalizable manner was effectively evaluated, closely mirroring potential clinical deployment scenarios.
3. Reinforcement Learning Pipeline
The core architecture of the proposed system is visually summarized in Figure 1: Depression Treatment Simulation and Optimization Flowchart. This pipeline systematically integrates synthetic data simulation, feature engineering, deep reinforcement learning, and real-time recommendation generation to build a fully operational decision support system for depression treatment.
The process begins with synthetic data generation, where a simulated patient population is created to capture the complex variability seen in clinical depression. This dataset is enriched through feature engineering to provide temporal and behavioural context, including rolling symptom averages and lag features that track short- and long-term symptom trends. Following feature preparation, the data is partitioned using a train-test split at the patient level, ensuring that the reinforcement learning agent is evaluated on completely unseen patient trajectories. This split is critical to prevent data leakage and to accurately assess the agent’s generalization performance to new patients. The reinforcement learning environment simulates patient trajectories over time. Within this environment, the Deep Q-Network (DQN) agent interacts sequentially with the patient’s state by selecting actions that represent treatment changes, dosage adjustments, or continued medication use. The agent’s learning process is supported by three key components:
Experience Replay: Stores past experiences and randomly replays them to improve learning stability.
Target Model Update: Periodically updates a separate target network to stabilize Q-value estimations.
Epsilon-Greedy Exploration: Balances exploration of new strategies and exploitation of learned policies by gradually reducing random action selection over time.
After training, the system proceeds to model evaluation using the test set, followed by deployment into a real-time recommendation system capable of processing new patient entries and dynamically generating treatment suggestions.
Figure 1. Depression treatment simulation and optimization flowchart.
The flowchart in Figure 1 illustrates the complete learning and deployment pipeline from synthetic data generation to real-time recommendation. The modular architecture detailed in Figure 1 ensures that the proposed system is scalable, interpretable, and ready for clinical application in precision psychiatry.
To facilitate reproducibility and transparency, the full specification of the agent’s state representation, action space, and key hyperparameters is outlined. The state vector included daily values for five core symptoms (mood, sleep, energy, concentration, appetite), the 7-day moving average of symptom scores, medication adherence percentage, time on current medication, and one-hot encoding of the previous action. The action space consisted of six discrete options: no treatment change or switching to one of five commonly used antidepressants (Sertraline, Escitalopram, Fluoxetine, Venlafaxine, Bupropion). The Deep Q-Network (DQN) was implemented as a three-layer multilayer perceptron with hidden layers of 128, 64, and 32 units, respectively, using ReLU activations. The network was trained using a learning rate of 0.001, a replay buffer of 50,000 transitions, and a batch size of 64. The target network was updated every 10 episodes, and exploration followed an epsilon-greedy schedule decaying from 0.7 to 0.01 across 300 episodes. These architectural and learning choices were empirically selected to ensure policy stability and learning efficiency in a temporally complex treatment environment.
4. Results
4.1. Data Visualization
The initial analysis of the synthetic depression dataset and the simulated patient responses is presented in Figure 2, which contains four key visualizations arranged in a 2 × 2 grid.
Figure 2. Data visualization overview.
Top-Left: Symptom Severity Distribution
The top-left plot illustrates the distribution of daily symptom severity scores across the entire synthetic patient population. Most symptom scores are concentrated below 0.4, indicating that most simulated patients experienced mild to moderate depression symptoms. This skewed distribution is consistent with real-world clinical datasets, where extreme severity is less frequently observed. The plot’s bell-shaped curve confirms that the synthetic symptom trajectories reasonably capture the expected variability seen in clinical practice.
Top-Right: Medication Distribution
The top-right plot presents the medication distribution over the total number of treatment days. Interestingly, the plot reveals that most recorded days correspond to “No Medication”. This could be due to early simulation days before medication initiation, stable periods where no treatment changes were required, or the agent frequently recommending no change based on patient status. The dominance of the “None” category suggests that, like real clinical workflows, medication changes are conservative and carefully timed, reflecting the agent’s learned behaviour.
Bottom-Left: Treatment Change Probability Over Time
The bottom-left plot depicts the treatment change probability as a function of days on medication. The probability remains consistently near zero across the timeline, indicating that treatment changes were rare in the simulated dataset. This aligns with the model’s design, where medication switches and dose adjustments are probabilistically constrained to occur only under specific symptom patterns or poor responses. The low change rate may also reflect a cautious treatment policy, resembling real-world clinical conservatism.
Bottom-Right: Patient Symptom Trajectory
The bottom-right plot showcases an individual patient’s (Patient 0) symptom severity trajectory over the entire simulation period. The plot clearly visualizes fluctuating symptom patterns, with occasional sharp spikes and drops, indicating the influence of random life events, treatment effects, or potential medication changes. Red vertical lines (if plotted) would typically highlight treatment change points, further validating the dynamic and responsive nature of the simulation. This trajectory demonstrates the complex, day-to-day variability the reinforcement learning agent must manage when optimizing treatment decisions.
Top-Left: Symptom Severity Distribution, Top-Right: Medication Distribution, Bottom-Left: Treatment Change Probability Over Time, Bottom-Right: Symptom Trajectory of Patient 0.
4.2. Agent Training Performance
The reinforcement learning agent’s training progress is visualized in Figure 3, which presents two key performance indicators: total reward per episode and epsilon decay over the training timeline. These visualizations are arranged side by side for direct comparison and interpretation.
Left Plot: Reward Per Episode
The left plot in Figure 3 illustrates the agent’s total reward trajectory across 300 training episodes. The reward curve shows notable fluctuations in the early episodes, indicating the agent’s exploration of various treatment actions and sequences. As training progresses, the reward pattern gradually stabilizes, reflecting the agent’s increasing ability to select more effective treatment strategies. The oscillations in reward are expected due to the complexity of the environment and the stochastic nature of symptom responses and medication effects. Importantly, the absence of significant downward trends suggests that the agent is successfully learning a balanced treatment policy without severe penalty accumulation over time.
This visual trend confirms that the agent is progressively refining its decision-making strategy, improving its capacity to recommend treatment actions that optimize long-term patient outcomes within the simulated environment.
Right Plot: Epsilon Decay
The right plot in Figure 3 displays the epsilon decay curve, which represents the agent’s exploration rate throughout training. Initially, the agent operates under a high epsilon value (~0.7), promoting random exploration of actions to adequately sample the state-action space. As training advances, epsilon decays rapidly and asymptotically approaches its minimum threshold near zero. This decay confirms that the agent systematically transitions from an exploration-heavy phase to an exploitation-driven policy, focusing on actions that yield the highest expected rewards based on prior learning. This behaviour is essential for reinforcement learning agents to first discover diverse strategies and later commit to the most successful ones, ensuring optimal long-term performance [42]-[45].
Figure 3. Agent training performance.
Left: Reward per Episode, showing stability and learning progression. Right: Epsilon Decay, illustrating the shift from exploration to exploitation during training.
4.3. Evaluation Results
Following agent training, the system was rigorously evaluated on unseen synthetic patients to assess its decision-making consistency and policy generalization [46]-[48]. The evaluation results are presented in Figure 4 and Figure 5, which capture the agent’s reward distribution and action selection patterns.
The top plot in Figure 4 depicts the distribution of immediate rewards collected by the agent during the evaluation phase. The distribution closely approximates a normal curve centred around zero, indicating that the agent achieved a balanced trade-off between symptom improvement and incurred penalties (such as unnecessary treatment changes). The concentration of rewards near zero suggests that the agent generally maintained symptom stability without introducing frequent, destabilizing adjustments [49]-[51]. Positive rewards reflect successful symptom reductions, while negative rewards represent instances where the agent’s decisions may not have led to symptom improvement. The relatively symmetric spread around zero further confirms that the agent’s learned policy was neither overly aggressive nor excessively conservative in its treatment recommendations [52]-[54]. This balanced reward profile is a critical outcome in clinical decision-making scenarios, where over-treatment or under-treatment can both have adverse consequences. The bottom plot in Figure 5 illustrates the agent’s action selection frequencies across the evaluation dataset. Actions are categorized into ‘No Change’ (action 0) and five distinct medication options (actions 1 to 5). The distribution shows that the agent utilized all available actions with relatively uniform frequency, indicating that the agent did not overfit to a single medication or default excessively to the “No Change” strategy. This action diversity suggests that the agent learned a well-balanced policy, capable of selecting different medications and appropriately maintaining or changing treatments based on patient-specific contexts. To benchmark the performance of the reinforcement learning agent, two baseline policies were implemented for comparison. The first was a random policy that selected actions uniformly across the six options, while the second was a rule-based policy that automatically switched medications every six weeks regardless of symptom status. Evaluation was performed on a held-out set of unseen synthetic patients. The RL agent achieved a mean symptom reduction of −0.23, compared to −0.08 and −0.15 for the random and rule-based baselines, respectively. Additionally, the RL agent executed an average of 2.1 medication switches per episode, which was substantially lower than the 4.8 and 6.0 observed in the baseline policies, indicating more conservative and targeted treatment modifications. In terms of reward, the RL agent achieved an average episode reward of +0.94, outperforming both baselines. These results confirm that the learned policy provides superior symptom relief with fewer treatment disruptions, supporting its clinical plausibility and decision efficiency.
![]()
Figure 4. Reward distribution during evaluation.
The agent’s reward profile is centred around zero, indicating balanced decision-making between benefits and penalties.
Figure 5. Action distribution during evaluation.
The agent selected “No Change” and each medication option uniformly, demonstrating exploration of the entire action space and balanced policy learning.
The evaluation results confirm that the agent generalizes well to new patient profiles, maintaining a dynamic and adaptable treatment policy without bias towards any particular action. This highlights the system’s potential as a clinically useful, flexible decision support tool for depression management.
5. Real-Time Recommendations
The real-time treatment recommendation system was deployed to simulate practical, patient-specific decision-making [55]-[57]. This system integrates the trained reinforcement learning agent into a clinical workflow capable of processing sequential symptom inputs and generating treatment recommendations dynamically as new patient data becomes available. The system was evaluated using a hypothetical patient example to demonstrate its decision logic and responsiveness to evolving patient conditions. In this example, the system provided two consecutive treatment recommendations based on the patient’s symptom profiles at different time points.
First Recommendation: The agent’s initial output was a recommendation to “Switch to Escitalopram.”
This decision suggests that, based on the presented symptom severity and treatment history, the agent identified the patient’s current treatment regimen (in this case, Sertraline) as ineffective or suboptimal. The recommendation to switch medications aligns with clinical best practices, where a lack of symptom improvement after an adequate trial period warrants a reassessment and potential medication change. This demonstrates the agent’s capacity to detect suboptimal responses and initiate timely adjustments.
Second Recommendation: At the patient’s subsequent visit, after the medication switch, the agent provided a “No Change” recommendation.
This response indicates that the patient’s symptom trajectory had stabilized, and further intervention was not immediately necessary. Clinically, this reflects an appropriate maintenance strategy, where ongoing monitoring is preferred over unnecessary medication adjustments when symptom control is achieved.
Clinical Alignment: The agent’s behaviour mirrors human clinical reasoning: Adjust treatments when symptoms persist or worsen despite current therapy. Maintain stability when symptom levels suggest treatment effectiveness. This dynamic and context-aware decision-making illustrates that the reinforcement learning system not only performs well in a simulated environment but also has the potential to support clinicians in real-time scenarios by providing personalized, evidence-driven treatment guidance. The system’s ability to adapt treatment recommendations based on up-to-date patient data, while avoiding over-treatment, positions it as a valuable tool for advancing precision psychiatry and improving patient outcomes in depression management.
6. Discussion
6.1. Clinical Implications
This study presents a fully operational Reinforcement Learning (RL) framework capable of providing AI-driven, patient-specific treatment recommendations for depression management. By simulating diverse patient trajectories using synthetically generated data, the system offers a privacy-preserving alternative to traditional clinical datasets, enabling extensive model development without compromising patient confidentiality. The model’s ability to dynamically adjust treatment plans based on real-time symptom inputs reflects its potential to significantly enhance clinical decision support. This is especially relevant in depression treatment, where medication response varies substantially across individuals, and timely intervention is critical to improving patient outcomes [58]-[61]. The system’s real-time functionality positions it as a practical tool for precision psychiatry, potentially reducing the time clinicians spend on trial-and-error prescribing.
6.2. Strengths
The proposed framework demonstrates several key strengths:
Privacy-Preserving, Fully Synthetic Pipeline: The exclusive use of synthetic data enables large-scale training and testing while fully safeguarding patient identities and mitigating ethical concerns related to data sharing.
Robust Deep Q-Learning Implementation: The agent employs experience replay and target model updates, which are essential for stable learning in complex, time-dependent environments. The system effectively balances exploration and exploitation through epsilon decay, ensuring both sufficient policy discovery and convergence.
Real-Time, Patient-Specific Recommendations: Unlike static models, the deployed agent can continuously process new symptom data and provide timely, individualized treatment guidance in an adaptive manner.
6.3. Limitations
Despite its promise, the system has inherent limitations:
Synthetic Data Generalization: Although the synthetic dataset was carefully designed to mimic real-world clinical scenarios, it may not fully capture the multifactorial complexity of actual patient responses, comorbidities, and social factors influencing treatment outcomes.
Sensitivity to Reward Design: The agent’s learning heavily depends on the structure of the reward function. An improperly weighted reward system may lead to undesirable treatment strategies, such as excessive medication switching or unwarranted dose stability.
6.4. Future Work
To further enhance the system’s clinical applicability and decision robustness, several future directions are proposed:
Validation on Real-World Clinical Datasets: Applying the trained agent to actual patient data will be essential to verify its clinical utility, reliability, and generalization outside the synthetic domain.
Integration of Multimodal Data: Future iterations should incorporate additional patient information, including genetic markers, behavioural metrics, and environmental factors, to more accurately model depression heterogeneity and treatment response variability.
Exploration of Advanced RL Architectures: Investigating alternative reinforcement learning paradigms, such as Actor-Critic models or Proximal Policy Optimization (PPO), could improve learning efficiency, stability, and policy flexibility, particularly in high-dimensional clinical environments. By addressing these areas, the system could evolve into a clinically viable decision support tool capable of assisting healthcare professionals in delivering more effective, timely, and personalized depression care.
7. Conclusion
This study introduces a novel Reinforcement Learning (RL) framework designed to optimize antidepressant treatment pathways through the use of fully synthetic patient data. By leveraging a carefully constructed simulation environment that captures temporal symptom patterns, medication effects, adherence variability, and life events, the system effectively trains a deep Q-learning agent capable of making dynamic, patient-specific treatment decisions. The reinforcement learning agent demonstrated the ability to recommend appropriate treatment adjustments when symptom improvement was insufficient and to maintain treatment stability when patient conditions stabilized. This behaviour closely aligns with clinical best practices, where timely medication changes and careful maintenance are both essential components of effective depression management [62]-[64]. A key strength of this framework is its real-time adaptability. Unlike traditional static models, the deployed system can continuously process sequential patient data and provide immediate, personalized recommendations. This supports the goal of moving toward precision psychiatry, where treatments are tailored to individual patient trajectories rather than relying on generalized population averages. Importantly, the use of synthetically generated data ensures that the entire training and testing pipeline remains privacy-preserving and ethically sound, while still providing sufficient diversity and complexity for the agent to learn robust policies. Overall, this research contributes a scalable, explainable, and data-efficient approach to AI-driven decision support in mental health care. While further validation on real-world datasets is necessary to confirm clinical applicability, the system offers a promising foundation for enhancing depression treatment through reinforcement learning and intelligent automation.
Conflicts of Interest
The authors declare no conflicts of interest.