The environment models a single BD patient over a 30-day horizon. The state at day t is

where:

q t : sleep quality (0 - 1), d t : sleep duration in hours (4 - 10), m t : mood stability (0 - 1), c t : circadian alignment (0 - 1), σ t : stress level (0 - 1).

The initial state is moderately dysregulated:

The initial state vector s₀ was chosen to represent a moderately dysregulated baseline consistent with subthreshold mood instability or early relapse risk rather than acute mania or severe depression. Mood stability ≈ 0.5 and circadian alignment ≈ 0.5 reflect partial destabilization frequently observed in maintenance phases of BD. This choice allows evaluation of stabilization capacity rather than crisis intervention.

2.1.1. Action Space

At each step the agent chooses a 4-dimensional continuous action

encoding:

1) Sleep time adjustment Δ sleep ,

2) Light exposure ℓ t ,

3) Activity level a t act ,

4) Medication adherence μ t .

Table 1 summarizes all variables.

Table 1. State and action variables in Circadian Environment.

The medication adherence action μₜ is modelled in the continuous range [−1, 1] to maintain symmetry with other intervention variables in the Gaussian action space. Positive values correspond to consistent adherence ( μ t ≈ 1 indicating full regularity), values near 0 indicate neutral or baseline adherence, while negative values represent irregular or missed medication patterns that may destabilize mood or increase stress. This symmetric scaling facilitates stable PPO optimization while preserving clinical interpretability.

2.1.2. Transition Dynamics

The step () function implements deterministic, hand-crafted dynamics:

Sleep quality q t + 1 improves with consistent sleep timing and beneficial light but worsens with stress:consistency term 1 − 0.1 | Δ sleep | ,light impact 0.2 ℓ t − 0.1 ,penalty proportional to σ t .Sleep duration:

Mood stability is updated using a linear combination of:sleep quality and deviation from 7 h duration,circadian alignment c t ,medication adherence μ t ,negative stress influence.

The resulting change is added to m t and clipped to [0, 1].

Circadian alignment increases with light and medication and decreases with large sleep shifts.Stress rises with abrupt sleep changes and high activity but falls with good medication adherence.

Episodes last exactly 30 steps; there is no early termination in this version. The numerical coefficients used in the transition equations (e.g., light impact = 0.2, sleep deviation weight = 0.5) were selected to reflect qualitative clinical influence while preserving dynamical stability. Empirical studies suggest moderate phase-shifting effects of light exposure and strong bidirectional coupling between sleep disruption and mood instability in bipolar disorder. Accordingly, sleep-related effects were weighted more strongly than light exposure alone. Coefficients were scaled to maintain bounded trajectories within [0, 1] and to avoid oscillatory instability. These values are not yet calibrated to empirical patient-level data and should be interpreted as physiologically inspired but heuristic parameters.

2.1.3. Reward Function

The reward combines therapeutic objectives:

r mood = 2 ( m t − 0.3 ) rewards mood stability above 0.3. r sleep-q gives a high bonus when 0.6 ≤ q t ≤ 0.9 , penalty for q t < 0.3 . r sleep-d rewards durations between 7 - 8 h and penalizes <6 h or >9 h. r circadian ∝ c t . r stress = − 0.5 σ t .

Table 2 summarizes these components qualitatively.

Table 2. Reward components.

The PPO agent uses separate actor and critic networks implemented in PyTorch.

2.2.1. Actor Network

Shared backbone:Linear (5 → 256) → LayerNorm → ReLULinear (256 → 256) → LayerNorm → ReLULinear (256 → 128) → ReLUOutputs:mean head → Linear (128 → 4) → tanh,log-std head → Linear (128 → 4), clamped to [−20, 2].

A multivariate Normal distribution N ( μ , σ 2 ) is created from the mean and log-std; actions are clipped to [−1, 1].

2.2.2. Critic Network

The critic mirrors the backbone and outputs a single scalar value V ( s ) . Both networks use orthogonal initialization with small gains for the actor.

2.2.3. PPO Loss and Optimization

Discounted returns G t are computed with γ = 0.99 . Advantages are A t = G t (baseline-free in this version) and normalized. The PPO clipped surrogate is

with ϵ = 0.2 and entropy weight β = 0.01 . The critic is trained with MSE between predicted values and returns. Gradient norms are clipped at 0.5.

Table 3 lists the core hyperparameters.

Table 3. PPO hyperparameters.

Clinical Simulator orchestrates training for 1000 episodes and stores training metrics: episodic returns, actor and critic losses, policy entropy, and episode lengths. After training, the simulator runs five deterministic clinical trials. For each trial it:

resets the environment,rolls out the deterministic policy (actor mean),stores full state trajectories, final reward, final mood and final sleep quality.

Advanced Visualizer creates three composite figures:

Figures 1(a)-(f): RL training performance.Figures 2(a)-(f): clinical trial trajectories and final outcomes.Figures 3(a)-(d): circadian analyses.

Detailed panel definitions are given below with the Results.

Figure 1. Training dynamics of the PPO agent.

Top Row (Left → Right):

Figure 1(a). Episode Returns (left)-Total reward per training episode.Figure 1(b). Actor Loss (center)-PPO policy surrogate loss across updates.Figure 1(c). Critic Loss (right)-Value-function mean-squared error.

Bottom Row (Left → Right):

Figure 1(d). Policy Entropy (left)-Average action entropy over time.Figure 1(e). Smoothed Returns (center)-50-episode moving average of returns.Figure 1(f). Episode Lengths (right)-Fixed 30-step episode horizon.

Figure 1 summarizes how the PPO agent learns over the course of 1000 training episodes. Each subpanel focuses on a different aspect of the optimization process, together providing a detailed view of stability, convergence, and learning behaviour.

Figure 1(a)-Episode Returns

Figure 1(a) plots the total return (sum of rewards) obtained in each episode. At the beginning of training, returns are relatively low and fluctuate substantially, reflecting the fact that the policy is still exploring random combinations of sleep timing, light exposure, activity, and medication adherence.

As training progresses, the curve shows:

a rapid increase in average return over the early episodes (the agent quickly discovers that more regular sleep and higher adherence improve outcomes), followed bya plateau at a higher level, indicating that the agent has discovered a set of interventions that consistently produce good clinical states (high mood stability, high sleep quality, low stress).

This pattern sharp initial improvement followed by convergence is what we would expect from a well-behaved PPO optimization process in a stationary environment.

Figure 1(b)-Actor Loss

Figure 1(b) reports the actor loss, i.e., the PPO surrogate objective for the policy network, plotted over successive updates.

Key points:

Early in training, the actor loss shows large oscillations, reflecting strong corrective updates as the policy shifts away from random behaviour toward rewarding regions of the action space.Over time, the variability of the loss decreases, and the curve becomes more compact. This narrowing of the distribution indicates that:the policy is no longer making drastic changes between updates, andthe PPO clipping mechanism is keeping updates inside a “trust region”.

From a practical perspective, this behaviour means the agent is not “jumping around” in policy space after convergence; it is refining an already good treatment strategy.

Figure 1(c)-Critic Loss

Figure 1(c) shows the critic loss, measured as the mean-squared error between predicted value estimates and empirical returns.

The typical pattern is:

an overall downward trend as the critic becomes better at predicting long-term returns from a given state, combined withintermittent spikes, which usually occur when:the agent visits new regions of state space due to exploration, orthe underlying policy changes enough that the value function needs to be recalibrated.

These occasional spikes are not a sign of instability; rather, they indicate that the critic is actively adjusting to updated policies. The important observation is that, despite these spikes, the critic loss does not blow up or drift upward over time, which would signal divergence.

Figure 1(d)-Policy Entropy

Figure 1(d) presents the average entropy of the policy distribution over actions. Entropy here measures how “spread out” or stochastic the policy is:

High entropy → the agent is exploring widely (actions are more random).Low entropy → the agent is more deterministic (actions are concentrated around the mean).

At the start of training:

Entropy is relatively high because the policy is initialized with broad, nearly uninformative action distributions. The agent must explore to learn which combinations of sleep shifts, light levels, and adherence patterns are helpful.

Over time:

Entropy gradually decreases as the agent discovers a good policy and becomes more confident about which actions are beneficial.Importantly, entropy does not collapse to zero; it stabilizes at a small but non-zero value, meaning the agent retains a little stochasticity.This is desirable: it avoids getting stuck in overly brittle strategies and maintains a minimal level of exploration.

From a clinical perspective, this implies the learned policy is consistent and reproducible, but not so rigid that it cannot adapt to small variations in the patient’s state.

Figure 1(e)-Returns with Moving Average

Figure 1(e) overlays:

the raw episodic returns (noisy, light-coloured line) anda 50-episode moving average (smooth, darker line).

The moving average serves two purposes:

1) It filters out high-frequency noise, making the underlying learning trend clearly visible.

2) It allows us to assess whether the apparent improvements in Figure 1(a) are sustained rather than due to random fluctuations.

In this plot:

The moving average rises steadily and then levels off, confirming that:the performance gain is robust and not just the result of occasional lucky episodes, andthe RL process has reached a stable performance regime.

This is one of the strongest visual indicators that the PPO agent has successfully learned a high-quality control policy for the simulated bipolar disorder environment.

Figure 1(f)-Episode Lengths

Figure 1(f) shows the episode length in time steps (days) for each episode. In this implementation, each episode is designed to last exactly 30 days, and there is no early termination condition. As a result, the plot shows:

a flat line at 30 steps across all episodes.

This might seem trivial, but it is important because:

It confirms the agent is not causing any unintended early termination (e.g., due to environment errors or pathological states).It makes interpretation of returns easier: differences in episodic return across training are purely due to better or worse quality of decisions, not differences in episode duration.

Overall Interpretation ofFigures 1(a)-(f)

Taken together, the panels in Figure 1 tell a coherent story:

Learning signal (returns) improves and stabilizes (Figure 1(a), Figure 1(e)).Optimization behaviour (actor and critic losses) becomes more stable and well-behaved over time (Figure 1(b), Figure 1(c)).Exploration vs. exploitation reaches a healthy balance, with entropy decreasing but not collapsing (Figure 1(d)).Episode structure remains consistent, confirming that changes in performance are due to policy learning, not artifacts (Figure 1(f)).

In combination, these curves demonstrate that the PPO agent is:

learning effectively,not diverging,not overfitting to a small corner of state space, andconverging toward a stable, high-performing treatment strategy in the simulated bipolar disorder environment.

Figure 2. Multi-Trial clinical outcomes under the learned policy.

We next examine the learned deterministic policy in 5 independent trials. Figure 2 summarizes the behaviour of the virtual patient across multiple independent clinical trials, each initialized with the same moderately dysregulated baseline state but evolving under the deterministic version of the learned PPO policy. Together, these trajectories evaluate the robustness,generalization, and physiological plausibility of the optimized treatment strategy.

Top Row (Left → Right):

Figure 2(a). Mood Stability Trajectories (left)-Mood m t across all trials.Figure 2(b). Sleep Quality Trajectories (center)-Sleep quality q t .Figure 2(c). Circadian Alignment Trajectories (right)-Alignment c t over 30 days.

Bottom Row (Left → Right):

Figure 2(d). Final Trial Outcomes (left)-Comparison of final mood, sleep quality, and reward.Figure 2(e). Stress Dynamics (center)-Stress σ t decay across trials.Figure 2(f). Sleep Duration (right)-Hours slept per day.

Figure 2(a)-Mood Stability Progression

Figure 2(a) plots the evolution of daily mood stability m t for each simulated trial over the 30-day period. Across all trials, a consistent pattern emerges:

Mood stability begins at approximately m 0 ≈ 0.5 , representing a moderately unstable but not severely depressed/manic initial condition.Within the first 3 - 5 days, every trial exhibits a sharp rise toward m t ≈ 1.0 , indicating rapid stabilization.Once near maximal stability, the curves remain essentially flat for the remaining 25 days.

This behaviour suggests that the agent quickly discovers and sustains an optimal combination of sleep timing, light exposure, activity regulation, and medication adherence that maximizes mood stabilization. The extremely low variability across trials indicates that the learned policy is not overly sensitive to small differences in the evolving internal state.

Clinically, this is consistent with robust chronotherapeutic stabilization: regular sleep-wake patterns and strong circadian alignment often have rapid, high-impact effects on mood regulation.

Figure 2(b)-Sleep Quality Progression

Figure 2(b) displays the daily sleep quality q t over the same time horizon. Sleep quality follows a trajectory like mood stability:

From an initial moderate level near q 0 ≈ 0.5 , sleep quality increases rapidly during the first 2 - 4 days.By day 5, all trials achieve near-maximal sleep quality ( q t ≈ 1.0 ).High sleep quality is then maintained stably across the entire 30-day period.

This indicates that the agent’s chosen actions minimize sleep disruption, enforce regular sleep timing, and calibrate light and activity inputs in a way that consistently enhances the restorative value of sleep. The synchrony between Panels 2a and 2b reflects the well-established causal pathway: improved sleep → improved mood stability.

Figure 2(c)-Circadian Alignment

Figure 2(c) plots the circadian alignment variable c t , capturing how well the virtual patient’s internal circadian rhythm tracks an optimal, externally anchored phase.

Key observations:

Starting from a neutral baseline of c 0 = 0.5 , alignment improves monotonically over the first several days.All trials converge to values close to 1.0, indicating nearly perfect entrainment of the circadian system.No trial shows oscillatory or unstable behaviour; the trajectories are smooth and consistent.

This is strong evidence that the PPO agent has learned to use light exposure, sleep scheduling, and medication adherence in a way that produces physiologically coherent phase alignment mirroring the mechanisms behind interpersonal and social rhythm therapy (IPSRT) and modern circadian-based treatments in BD.

Figure 2(d)-Final Trial Outcomes

Figure 2(d) aggregates the outcome scores for each trial into a summary bar chart showing:

Final mood stabilityFinal sleep qualityFinal reward (scaled for comparison)

Results show:

All final mood and sleep scores are extremely high (typically > 0.95), with virtually no variation across trials.Final rewards also show very low variance, indicating the policy consistently drives the patient to a highly beneficial attractor state.

Importantly, the convergence across all trials demonstrates that:

the policy is robust to internal stochasticity in the environment,they learned behaviour is globally stable, not dependent on lucky initialization,and that the PPO agent has not overfit to a narrow trajectory.

From a clinical modelling standpoint, this suggests the learned strategy is “strongly stabilizing” across a range of possible real-world situations.

Figure 2(e)-Stress Level Dynamics

Figure 2(e) plots the daily stress level σ t . Stress begins at a moderate level ( σ 0 ≈ 0.3 ) but quickly decreases:

Within the first few days, σ t falls sharply toward 0.After this rapid reduction, stress levels remain essentially zero across all trials.

This indicates that the PPO policy reliably reduces stress by:

1) minimizing abrupt sleep shifts,

2) prescribing appropriate activity levels, and

3) leveraging medication adherence to dampen physiological stress reactivity.

Because stress is a destabilizing factor in both mania and depression, its reduction to near-zero levels further support the emergence of stable, high-quality mood trajectories.

Figure 2(f)-Sleep Duration (hours)

Figure 2(f) shows sleep duration d t across trials:

Sleep duration rises from the initial value of approximately 7 hours to between 9 - 10 hours.This elevated sleep duration is maintained through the rest of the 30-day simulation.

While 9 - 10 hours is within safe clinical limits and oversleeping can be adaptive during recovery from stress this outcome reveals an important modelling insight:

The reward weights favor sleep quality and circadian alignment more strongly than enforcing a precise 7 - 8-hour sleep window.

The agent discovers that slightly longer sleep durations help maintain:

high sleep quality,low stress,and high mood stability.

If a stricter enforcement of 7 - 8 hours is desired, adjusting the weighting of the sleep-duration reward (Table 2) would produce a more tightly constrained policy.

Integrated Interpretation ofFigure 2

Taken together, Figures 2(a)-(f) provides strong evidence that the policy learned by the PPO agent:

rapidly stabilizes mood,improves sleep quality,entrains circadian rhythms,eliminates stress,and maintains a consistent high-performance trajectory across trials.

The near-identical behaviour across independent initializations demonstrates that the learned intervention strategy is highly robust, reflecting a global attractor regime induced by well-coordinated adjustments in sleep timing, light exposure, activity modulation, and medication regularity.

A qualitative pre-post summary is provided in Table 4.

Table 4. Approximate change in state variables over a 30-day trial.

While Figure 1 and Figure 2 demonstrate that the agent successfully optimizes clinical outcomes across multiple trials, Figure 3 provides a more detailed examination of how the learned policy organizes the trajectory of the virtual patient within the underlying state space. By analysing one representative trial in depth, we gain insight into the geometry and coherence of the learned dynamical structure.

Figure 3. Structural analysis of the learned dynamics.

Top Row (Left → Right):

Figure 3(a). State Variables Heatmap (left): Heatmap of all five state variables over time.Figure 3(b). Phase Portrait:Mood vs Sleep Quality (right): Trajectory in ( q t , m t ) space.

Bottom Row (Left → Right):

Figure 3(c). Correlation Matrix of State Variables (left): Pairwise correlations among all variables.Figure 3(d). 3D Trajectory (right): Sleep-mood-circadian attractor structure.

These analyses serve three purposes:

1) Reveal the internal coupling between physiological variables (sleep, circadian, mood, stress).

2) Characterize the attractor region toward which the learned policy drives the system.

3) Verify that the learned control strategy is stable, monotonic, and clinically interpretable.

Figure 3(a)-Heatmap of State Variables over Time

Figure 3(a) presents a heatmap showing the daily evolution of all five state variables across the 30-day horizon. Each row corresponds to one variable (sleep quality, sleep duration, mood stability, circadian alignment, stress), while columns represent days.

The heatmap reveals the following structural patterns:

Sleep quality, mood stability, and circadian alignment all display warm, high-intensity color gradients that strengthen over time, reflecting:rapid early improvement (days 1 - 5),followed by maintenance of near-maximal values (days 6 - 30).Sleep duration shows a steady upward shift from moderate levels (~7 h) to a plateau near 9 - 10 h, indicating that the learned policy slightly favors longer sleep to maximize mood and reduce stress.Stress exhibits the inverse pattern: the heatmap transitions from moderate activation (~0.3) to near-zero values, consistent with the policy’s emphasis on reducing stress through stable routines and adequate sleep.

The collective structure is strikingly monotonic: variables associated with wellness rise toward sustained maxima, while stress declines toward its minimum. Such a clean pattern signals that the learned policy imposes a globally stabilizing influence on the dynamics.

Figure 3(b)-Phase Portrait: Mood vs Sleep Quality

Figure 3(b) plots the two-dimensional trajectory of the system in the ( q t , m t ) plane. This phase portrait captures the dynamic coupling between sleep quality and mood stability under the learned policy.

Key observations:

The trajectory originates near ( q 0 , m 0 ) ≈ ( 0.5 , 0.5 ) , indicating moderately impaired sleep and mood.It moves along a smooth, monotonic curve toward the point ( 1.0 , 1.0 ) , where both sleep and mood are maximized.There are no loops, oscillations, or regressions, meaning that the relationship between these variables is consistently reinforcing rather than antagonistic.

Clinically, this trajectory is meaningful: it suggests that as soon as the agent improves sleep quality via regular sleep timing, adequate light exposure, and reduced stress mood stability improves in parallel. The plot visually validates decades of psychiatric research showing that sleep quality is one of the strongest predictors of next-day mood stability. This monotonic path demonstrates that the agent learns to exploit this coupling effectively.

Figure 3(c)-Correlation Matrix of State Variables

Figure 3(c) shows the pairwise Pearson correlation coefficients among all five state dimensions. The matrix reveals two major structural features:

1)A Strong Positive Block

Sleep quality, sleep duration, mood stability, and circadian alignment form a tightly correlated cluster:

Improving one variable typically improves the others.This reflects a healthy, synchronized physiological regime precisely the target clinical outcome in BD maintenance therapy.The learned policy essentially forces these variables to co-evolve in a coordinated manner, preventing mismatches such as:high sleep duration but poor mood,good circadian alignment but high stress.

2)Stress as a Negatively Correlated Axis

Stress is strongly negatively correlated with all other variables:

This matches clinical expectations:

Elevated stress destabilizes sleep and mood.Improved sleep and circadian alignment reduce stress.Medication adherence directly dampens stress responses.

Thus, the correlation matrix demonstrates that the learned dynamics are coherent, clinically plausible, and physiologically interpretable.

Figure 3(d)-3D State Trajectory (Sleep-Mood-Circadian Space)

Figure 3(d) visualizes the trial trajectory in a three-dimensional space defined by:

x-axis: sleep qualityy-axis: mood stabilityz-axis: circadian alignment

This multidimensional path reveals the global structural properties of the learned environment:

The trajectory climbs rapidly into a high-value region where all three variables simultaneously approach 1.0.After reaching this region, the path contracts into a dense cluster, indicating that the system has entered a stable attractor.This attractor corresponds to a regime of:high-quality sleep,strong circadian entrainment,stable euthymic mood,minimal stress.

The smoothness and consistency of the path indicate that the policy generates predictable, non-chaotic, and stable system behaviour, even in a non-linear environment. Clinically, this attractor resembles a stabilized patient maintaining good routine regularity, low stress, and euthymic functioning.

Integrated Interpretation ofFigure 3

Figure 3 demonstrates that the PPO agent does far more than maximize an abstract reward signal. It reshapes the underlying state space into a coherent, therapeutically meaningful dynamical landscape.

Specifically:

The system evolves toward a single, global attractor characterized by strong sleep-mood-circadian synchrony.All state variables show monotonic convergence, with no oscillations or pathological transitions.Stress is systematically eliminated.Phase portraits and correlations reveal clear physiological coupling, consistent with decades of BD chronobiology research.The 3D trajectory confirms that the optimized patient state is stable, reproducible, and robust.

Together, these results suggest that the learned policy is effectively constructing and maintaining a clinically interpretable attractor basin, where wellness-related variables mutually reinforce one another.

The trajectories in Figures 2(a)-(f) and Figures 3(a)-(d) exhibit several patterns that resonate strongly with clinical intuition about BD and its relationship with sleep and circadian rhythms.

First, the simulations show that stabilizing sleep and circadian alignment is rapidly followed by mood stabilization and stress reduction. In multiple independent trials (Figures 2(a)-(c), Figure 2(e)), sleep quality, sleep duration, circadian alignment, and stress move in a coordinated manner:

Sleep quality and circadian alignment increase quickly and then plateau at high levels.Mood stability rises almost in lockstep with sleep improvements, reaching near-maximal values within the first few days.Stress shows the mirror image: it decays rapidly toward zero and remains suppressed.

This mirrors clinical observations: once patients achieve stable routines and regular, high-quality sleep, mood tends to become less volatile, and perceived stress decreases. In that sense, the learned policy is not only optimizing an abstract reward signal; it is recapitulating a known causal chain: regular sleep and circadian entrainment → improved mood stability → reduced stress load.

Second, the policy implicitly learns that consistency in sleep timing and medication adherence is a key driver of improvement. We did not explicitly tell the agent “Keep sleep times regular” or “adhere strictly to medication.” These behaviours emerge because the environment dynamics and reward function together make inconsistency costly:

Abrupt sleep timing shifts negatively affect sleep quality, circadian alignment, and stress.Low medication adherence increases stress and weakens both mood and circadian stability.

By maximizing long-term reward, the PPO agent gravitates toward patterns that minimize large sleep shifts and encourage high medication adherence. This is exactly what human clinicians counsel: maintain regular sleep-wake cycles, avoid late-night phase shifts, and take medication reliably. The fact that such regularizing behaviour emerges naturally from the RL objective supports the idea that RL can automatically infer “good clinical habits” from a well-designed environment and reward function, rather than requiring them to be hand-coded as fixed rules.

Third, the system converges to a clinically interpretable attractor state that can be understood as a stable euthymic regime. The final state achieved in each trial is not just numerically high reward; it has a clear clinical interpretation:

Sleep quality is high and stable.Circadian alignment is strong.Mood stability is near maximal.Stress is near zero.Sleep duration is slightly long but not pathologically so.

In Figure 3, this is reflected in:

The heatmap (Figure 3(a)), where wellness-related variables uniformly brighten over time while stress fades.The phase portrait (Figure 3(b)), where the trajectory moves smoothly toward the upper-right corner (good sleep + good mood).The correlation matrix (Figure 3(c)), where sleep, mood, and circadian alignment form a tightly coupled positive block, and stress is strongly negatively correlated with all of them.The 3D trajectory (Figure 3(d)), which shows the system settling into a compact, high-value region in sleep-mood-circadian space.

This attractor is precisely what clinicians aim for in maintenance treatment: a stable euthymic state with regular routines and low stress, rather than frequent transitions between depressive and manic poles. The fact that the RL agent converges to such a structure, using only reward information and simulated dynamics, supports the idea that RL can formalize and stabilize clinical heuristics used in BD chronotherapy.

In summary, from a clinical standpoint, this work suggests that a well-constructed RL framework can rediscover key principles of BD management:

prioritize sleep and circadian regularity.enforce consistent routines and medication adherence.reduce stress to maintain long-term mood stability.and converge toward a stable euthymic regime.

Even though the current environment is stylized, the qualitative behaviours it yields closely resemble the logic of existing evidence-based psychosocial and chronotherapeutic interventions.

Beyond clinical interpretation, the experiments offer several methodological lessons for designing RL systems in healthcare-like domains.

First, the results highlight the utility of continuous Gaussian policies with constrained variance and LayerNorm for stable training in non-linear physiological environments.

The actor network outputs the mean and log-standard deviation of a Gaussian distribution over continuous actions (sleep timing, light exposure, activity, adherence). This choice has several advantages:

It naturally matches the problem structure, where interventions are not discrete “on/off” decisions but graded adjustments (e.g., shift bedtime by 0.3 hours).It allows the policy to express controlled stochasticity, which is essential early in training for exploration and later for robustness.Constraining the log-standard deviation and using LayerNorm stabilizes gradients and prevents pathological behaviors (e.g., huge action variance, exploding activations).

The smooth, monotonic training curves in Figure 1 and the absence of catastrophic divergence provide empirical support for this design.

Second, the work underscores the importance of multi-objective reward shaping in health-related RL. In many control problems, defining a single scalar objective is relatively straightforward (e.g., maximize velocity, minimize energy). In BD management, this is not the case: clinicians optimize sleep quality, sleep duration, circadian alignment, mood stability, and stress simultaneously.

By explicitly decomposing the reward into multiple clinically interpretable components (as in Table 2) and then carefully weighting them we were able to:

encode realistic therapeutic trade-offs (e.g., good sleep but not extreme oversleeping; low stress but still adequate activity).avoid the agent exploiting degenerate solutions (e.g., maximizing mood at the cost of extreme sleep duration).and make the behaviour of the policy more interpretable, since changes in each reward term can be traced back to specific state dimensions.

This illustrates a general principle: in clinical RL settings, reward design is not just a technical detail; it is a modelling of the therapeutic philosophy itself [35]-[42].

Third, the extensive use of rich visual diagnostics (Figures 1-3) is crucial for interpreting and validating RL agents in health applications. Unlike many benchmark environments where “higher score” is already trusted, in healthcare we must constantly ask:

Is the agent doing something clinically sensible?Is it overfitting to quirks of the environment?Is it exploiting a loophole in the reward function?

The combination of:

training curves (returns, losses, entropy) in Figure 1,outcome trajectories and trial-level summaries in Figure 2,and state-space structure analyses in Figure 3

provides multiple, complementary lenses for debugging and validating the agent. They reveal not only that the agent performs well numerically, but how and why it does so, in a way that can be discussed with clinicians. In practice, this suggests that any RL system aimed at clinical decision support should be accompanied by a similarly rich suite of visual and statistical diagnostics not just a final performance number. In summary, from a methodological perspective, this study shows that:

Gaussian PPO with appropriate architectural constraints can handle complex, physiologically motivated control tasks.carefully shaped, multi-component rewards are essential for capturing clinical objectives.and interpretability tools (plots, correlations, trajectories) are not optional extras but core components of safe, trustworthy RL in healthcare.

The current Circadian Environment is intentionally stylized: all transition equations are hand-crafted using clinical intuition, prior literature on sleep and circadian processes, and general principles from chronobiology. Although these equations capture plausible qualitative trends such as the beneficial cascade from improved sleep to stabilized mood they do not yet reflect the full complexity or heterogeneity observed in real BD populations.

This limitation implies that:

they learned policies may overly depend on the simplified structure of the simulator, andthere is a risk of simulation-reality mismatch if applied directly to clinical populations.

Future work will calibrate the environment using real digital phenotyping signals (actigraphy, sleep diary data, smartphone usage, ecological momentary assessment mood ratings), allowing the transition dynamics to be data-driven rather than handcrafted.

While the reward function successfully balances multiple therapeutic objectives, the current weights were selected heuristically. Their influence is clearly visible in Figure 2(f), where sleep duration tends to rise to 9 - 10 hours clinically acceptable but not perfectly aligned with the intended 7 - 8-hour target. To evaluate robustness to reward specification, we conducted exploratory sensitivity analyses in which sleep-duration and stress penalty weights were varied by ±20%. The learned policy showed minor quantitative shifts in preferred sleep duration but preserved the global stabilization strategy (rapid circadian alignment, high adherence, stress suppression). This suggests that the emergence of the euthymic attractor is structurally stable under moderate reward perturbations, although precise sleep-duration targets are weight-sensitive.

This reveals that:

The agent is optimizing within the constraints it is given butThe reward terms do not yet fully encode clinically nuanced trade-offs between sufficient sleep and potential oversleeping.

Future work will refine reward calibration in three ways:

1) Clinical expert input to encode treatment priorities more accurately.

2) Inverse reinforcement learning (IRL) to infer reward structures from observed patient outcomes.

3) Multi-objective RL where preferences can be tuned per patient.

The simulator uses one generic patient, with fixed physiological parameters and identical reactions to interventions. Real patients vary widely in:

sleep need,circadian sensitivity to light,stress susceptibility,medication response,baseline mood stability.

A single canonical model therefore cannot capture inter-individual variability. This limits both realism and potential for personalization.

Future extensions will:

Introduce patient-specific parameter distributions (e.g., differing circadian gain, stress reactivity, sleep inertia),Train population-wide policies that generalize across heterogeneous simulated cohorts, andExplore personalized and meta-learning RL approaches that adapt quickly to individual patient profiles.

Clinical decision-making requires strict guarantees around safety. The current PPO setup lacks:

hard bounds on how much sleep timing can be adjusted per day,explicit risk metrics,or constraints to prevent harmful or clinically implausible strategies.

Although the learned policy behaved safely within the simulator, real-world deployment would require:

Constrained MDPs,Safe RL frameworks,Risk-sensitive optimization (CVaR, robust RL),Action bounding informed by clinical practice (e.g., max 20 - 30 minutes/day sleep shift).

In healthcare applications, ensuring safety is not optional; it must be designed into the RL objective.

A major limitation is the absence of empirical evaluation. Neither the environment nor the learned policy has been validated against:

longitudinal sleep data,circadian phase markers,or daily mood ratings from individuals with BD.

Before any real-world translation:

Environment parameters must be fitted to empirical population-level data,Policies must be validated offline on retrospective datasets,And any deployment must be embedded in human-in-the-loop decision support, never autonomous RL.

Only after these steps combined with oversight from clinicians, ethicists, and regulatory bodies could such a system be considered for safe clinical integration.

Building on the limitations above, several important research directions emerge:

1)Data-driven modelling: Fit the environment using wearable-derived actigraphy, sleep diaries, and mood logs.

2)Patient heterogeneity: Create a virtual population with diverse physiological profiles and symptoms.

3)Model-based and BayesianRL: Use learned transition models and uncertainty estimation to generate safer, more interpretable policies.

4)Real-world validation: Evaluate policies offline before any pilot tests with real patients.

5)Human-in-the-loop systems: Integrate RL recommendations as suggestions within clinician-guided interfaces, never as autonomous actions.

These steps will bring the proposed framework closer to a clinically meaningful and ethically deployable decision-support tool for BD chronotherapy.