1. Introduction
Cancer remains one of the leading causes of mortality worldwide, with metastatic disease accounting for the majority of cancer-related deaths. Global estimates indicate a significant burden of cancer incidence and mortality across populations, highlighting the urgency for improved detection, monitoring, and treatment strategies [1]. Projections further suggest that cancer-related mortality will continue to rise due to demographic and epidemiological transitions [2].
Prostate cancer is among the most commonly diagnosed cancers in men and a major contributor to cancer-related deaths, particularly in its metastatic stage [3].
Prostate cancer progression varies widely, ranging from slow-growing localized tumors to aggressive metastatic disease. Screening methods such as prostate-specific antigen (PSA) testing have improved early detection, although their impact on long-term mortality remains debated [4].
At the molecular level, mutations in PSA-related genes and associated biomarkers have been linked to disease progression and reproductive health outcomes [5] [6]. Additionally, genetic alterations in key regulatory genes such as BRCA2, ATM, and CDK12 significantly influence tumor aggressiveness and metastatic potential [7].
Inflammation has also been identified as a critical factor contributing to prostate cancer initiation and progression, further complicating disease dynamics [8].
Several biological and lifestyle factors influence prostate cancer development and progression. Body mass index (BMI) and hormonal imbalances have been associated with reproductive health and may indirectly contribute to cancer risk [9].
The transition from localized to metastatic prostate cancer is a complex, multi-stage process involving genetic mutations, tumor microenvironment interactions, and systemic physiological changes. Long-term studies show that even patients under active surveillance for early-stage prostate cancer may eventually develop metastases, emphasizing the need for improved predictive models [10].
Metastatic prostate cancer presents significant clinical challenges due to its resistance to treatment and high mortality rates. Once cancer spreads beyond the primary site, therapeutic options become limited, and patient outcomes worsen substantially.
Understanding metastasis requires integrating biological, genetic, and clinical data. Recent work has explored stochastic and computational frameworks to model disease progression and treatment dynamics, offering new insights into metastatic behavior and patient-specific outcomes [11].
Various therapeutic strategies have been explored to manage prostate cancer progression and metastasis. Beyond conventional treatments, research has investigated bioactive compounds with anti-cancer properties.
Prostate cancer is a biologically complex malignancy characterized by progressive genetic, molecular, and metabolic alterations that drive tumor initiation, progression, and metastasis. Early molecular studies established the importance of androgen receptor signaling, oncogene activation, and tumor suppressor dysfunction, including PTEN and p53 abnormalities, in the development of prostate cancer and its clinical heterogeneity ranging from indolent to highly aggressive disease [13]. More comprehensive genomic analyses demonstrated that prostate cancer progression involves ETS gene fusions, DNA repair defects, PI3K pathway dysregulation, chromatin remodeling abnormalities, and clonal evolution contributing to therapeutic resistance and transition to Castration-Resistant Prostate Cancer [14]. Recent studies have additionally highlighted the role of metabolic reprogramming in metastatic dissemination, particularly within the bone microenvironment [15].
These findings suggest potential complementary therapeutic approaches, although further clinical validation is required.
Advances in computational modeling and public health informatics have introduced new possibilities for understanding and managing metastatic cancer. Data-driven approaches enable the integration of clinical, genetic, and epidemiological data to improve prediction, diagnosis, and treatment planning.
Emerging frameworks emphasize the importance of combining artificial intelligence with traditional medical knowledge to enhance healthcare delivery and patient outcomes [16].
Despite significant progress, several challenges remain in addressing metastatic prostate cancer:
Limited ability to predict metastasis accurately;
Variability in patient response to treatment;
Incomplete understanding of genetic and molecular drivers;
Need for personalized and adaptive treatment strategies.
Addressing these gaps requires interdisciplinary research combining biology, medicine, and computational science.
Metastatic prostate cancer remains a major global health challenge with significant mortality implications. While advances in screening, molecular biology, and treatment have improved understanding, effective management of metastatic disease remains limited.
Emerging computational approaches and novel therapeutic strategies offer promising directions for improving outcomes. Future research should focus on integrating biological insights with advanced modeling techniques to enable personalized and more effective treatment strategies.
In this study, we develop and analyze a nonlinear state space modeling framework for metastatic cancer progression. The proposed approach incorporates key biological features, including nonlinear growth kinetics, inter-site interactions, and treatment effects, while providing a systematic methodology for latent state estimation. Through simulation experiments designed to reflect clinically relevant scenarios, we evaluate the performance of several nonlinear filtering techniques and compare them to linear approximations. Our results demonstrate that nonlinear models yield improved estimation accuracy and better capture critical aspects of metastatic behavior.
The remainder of this paper is organized as follows. Section 2 presents the proposed nonlinear state space model and its biological interpretation. Section 3 describes the estimation methods, including the Extended Kalman Filter, Unscented Kalman Filter, and Particle Filtering. Section 4 provides simulation results and performance analysis. Section 5 discusses the implications of the findings for predictive oncology and clinical decision making, followed by concluding remarks in Section 6.
2. Nonlinear State Space Model Formulation
In this section, we present a nonlinear state space modeling framework for describing metastatic cancer progression. The model is designed to capture the evolution of latent tumor burden across multiple anatomical sites, while accounting for nonlinear biological processes such as growth saturation, metastatic spread, and treatment effects.
2.1. Latent State Representation
Let
denote the latent state vector at discrete time k, where each component
represents the tumor burden at site i, or a specific subpopulation such as treatment resistant cells. The dimension mmm depends on the number of metastatic sites or biological compartments considered. The latent state is not directly observable and must be inferred from clinical measurements. Its evolution is governed by nonlinear dynamics reflecting underlying biological mechanisms.
This representation enables a compact yet expressive description of metastatic disease by incorporating both spatial distribution and biological heterogeneity within a unified state vector. Since clinical measurements, including imaging and biomarkers, provide only indirect and noisy information about tumor burden, the latent state serves as an essential intermediary between biological processes and observed data. By modeling tumor evolution in this latent space, the framework allows for the integration of nonlinear growth dynamics, inter-site interactions, and treatment effects, while facilitating systematic estimation of disease progression through sequential inference methods.
2.2. Nonlinear State Dynamics
The temporal evolution of the latent state is modeled as
(1)
where
is a nonlinear function,
represents external inputs such as treatment, and
is process noise capturing unmodeled variability. In our model, Q is the process noise covariance matrix. Q captures everything the model does not perfectly explain, such as:
Biological variability in tumor growth;
Unmodeled interactions between metastatic sites;
Random fluctuations in cell proliferation.
Large Q corresponds to high uncertainty in tumor dynamics and aggressive or unpredictable disease. Small Q represents a more predictable evolution, stable or well characterized progression of the disease.
Example (prostate cancer):
Rapid metastatic spread to bones and other organs corresponds to larger Q;
Controlled prostate cancer (under therapy) corresponds to smaller Q. A biologically motivated form of the dynamics is given component wise by
(2)
where:
is the intrinsic growth rate at site i;
is the carrying capacity representing environmental constraints;
models metastatic transfer from site j to site i;
quantifies treatment efficacy;
is the treatment input at time k.
This formulation captures several key aspects of metastatic disease, including nonlinear growth saturation, inter-site coupling, and therapy induced suppression. Figure 1 shows simulated nonlinear metastatic tumor progression across three anatomical sites. Time evolution of latent tumor burden for three metastatic sites (Site 1, Site 2, and Site 3) over 100 discrete time steps, generated using a nonlinear state space model. Each trajectory reflects the combined effects of logistic growth, inter-site metastatic coupling, treatment response, and stochastic variability. Initially, all sites exhibit rapid growth driven by intrinsic proliferation rates, followed by a gradual slowdown as tumor burden approaches site specific carrying capacities, illustrating saturation effects characteristic of constrained biological systems. Differences in asymptotic levels across sites arise from heterogeneous parameters, including growth rates and carrying capacities.
![]()
Figure 1. Nonlinear metastatic tumor progression across three anatomical sites.
Inter-site interactions, modeled through coupling coefficients, contribute to the coordinated rise in tumor burden, particularly during the early and intermediate phases of progression. A constant treatment input is incorporated, producing a moderating effect on growth and contributing to stabilization at later time points. Small fluctuations observed around steady state levels are due to process noise, representing intrinsic biological variability and unmodeled dynamics. Overall, the figure demonstrates key features of metastatic disease progression, including nonlinear growth behavior, heterogeneous site dynamics, and the interplay between proliferation, spread, and treatment effects.
If the state vector is
(3)
then the tumor burden at time k is
(4)
2.3. Observation Model
Clinical measurements are typically indirect and noisy reflections of the underlying tumor burden. Let
denote the observation vector at time k. The observation model is defined as
(5)
where h(⋅) is a nonlinear measurement function and
. represents observation noise. A representative example is
(6)
where
reflects the contribution of each site to the measured data. This form captures nonlinear effects such as signal saturation or diminishing sensitivity at high tumor burden. Alternative observation models may be used depending on the measurement modality, including sigmoidal or rational functions to represent imaging or biomarker responses.
We may reconstruct the latent space by treating it as an inverse observation problem. Given
the observation does not identify each
separately from one scalar measurement
. It only gives information about the weighted sum of the latent states. Ignoring noise for a moment, exponentiating gives
(7)
With noise included, this becomes approximate:
(8)
This means the measurement constrains the latent state to a hyperplane in state space, not to a unique point. It is easy to prove this statement. Let us define (for a given observation):
(9)
For a fixed-time index k, the observation model reduces to a scalar constraint. This defines a hyperplane in the latent state space, implying that the hidden state is not uniquely identifiable from a single observation. There is only one equation and m unknowns. The problem is highly nonunique. The latent vector
is not uniquely identifiable unless we add more information.
We need at least one of these:
Multiple measurements at the same time;
System dynamics over time;
Structural assumptions such as sparsity or positivity;
Prior distributions on the latent state.
Figure 2 shows nonlinear observation sequence representing simulated prostate-specific antigen (PSA) measurements
, derived from the underlying tumor state vector
. The observation model maps the hidden tumor dynamics to clinically observable PSA levels through a nonlinear function h(⋅), while
represents stochastic measurement noise and biological variability. The resulting trajectory exhibits an initial rapid increase corresponding to tumor expansion, followed by a saturation phase reflecting limitations in tumor growth or PSA production. Small fluctuations around the plateau highlight the influence of noise and potential heterogeneity in tumor subpopulations, illustrating the imperfect and indirect relationship between PSA levels and true tumor burden.
Figure 2. Nonlinear observation sequence representing PSA measurements yk derived from the underlying tumor state Zk. The dynamics reflect logistic-like tumor growth with stochastic measurement noise.
Figure 3 shows the total metastatic tumor burden
over time, representing the aggregate tumor volume across three anatomical sites. The dynamics are governed by nonlinear growth processes, including logistic proliferation within each site and inter-site interactions. The curve demonstrates an initial phase of accelerated growth, followed by a transient adjustment period and eventual convergence to a quasi-steady state, indicating the effect of carrying capacity constraints and competitive interactions among tumor compartments. Minor fluctuations in the later stages reflect system noise and dynamic coupling between sites. This trajectory represents the underlying disease progression that gives rise to the observed PSA signal shown in Figure 2.
3. Reconstruction Using Particle Filtering
Particle filtering, also known as Sequential Monte Carlo methods, is a probabilistic framework for estimating the hidden states of nonlinear and non-Gaussian dynamical systems. Consider the state space model defined by the equations:
Figure 3. Total metastatic tumor burden over time, showing nonlinear growth dynamics with an initial rapid increase followed by a plateau and mild fluctuations.
(10)
The objective of particle filtering is to recursively estimate the posterior distribution
, where
denotes all observations up to time k. This distribution is approximated using a set of N weighted particles
and
, such that
(11)
where δ(⋅) is the Dirac delta function.
The algorithm proceeds recursively through prediction and update steps. In the prediction step, particles are propagated according to the state dynamics
(12)
incorporating process noise. In the update step, the particle weights are adjusted using the likelihood of the new observation
(13)
The weights are then normalized so that
(14)
To address particle degeneracy, a resampling step is introduced, where particles with low weights are discarded and particles with high weights are replicated. This ensures that computational resources are focused on regions of high posterior probability.
In the context of prostate cancer modeling, particle filtering enables the estimation of latent tumor states
from noisy PSA observations
. The nonlinear tumor growth dynamics and indirect measurement process make analytical solutions intractable, thereby motivating the use of particle-based approximations. This approach provides both state estimates and uncertainty quantification, which are essential for understanding disease progression and informing clinical decision making.
Figure 4 illustrates the time evolution of the true latent tumor burden and its corresponding estimate obtained using a particle filtering approach under a nonlinear state space framework. The blue curve represents the ground tumor burden, simulated using a logistic growth model with treatment effects and stochastic process noise. The orange curve shows the posterior mean estimate of the latent state inferred from noisy, nonlinear observations of the form
, which mimic PSA-like measurements commonly used in prostate cancer monitoring.
Figure 4. Particle filter estimation of latent tumor burden in a nonlinear prostate cancer progression model.
The shaded region corresponds to the 95% credible interval derived from the particle distribution, reflecting the uncertainty in the state estimate at each time step. Early in the time series, the uncertainty band is relatively narrow due to the strong influence of initial conditions and informative observations. As time progresses, uncertainty increases slightly, reflecting accumulated process noise and nonlinear effects, before stabilizing as repeated observations constrain the posterior distribution.
The close agreement between the estimated trajectory and the true latent state demonstrates the effectiveness of particle filtering in reconstructing hidden tumor dynamics from indirect and noisy measurements. Small deviations between the true and estimated trajectories are attributable to stochastic variability and the inherent limitations of finite particle approximations. Overall, the figure highlights the capability of sequential Bayesian inference to track nonlinear disease progression and quantify uncertainty in latent tumor burden over time.
4. Discussion
The results presented in this study demonstrate the effectiveness of a nonlinear state space modeling framework for capturing prostate cancer dynamics and their relationship to observable PSA measurements. The simulated tumor burden exhibits characteristic logistic-like growth, with an initial phase of rapid expansion followed by a gradual transition to a steady state. This behavior is consistent with biological expectations, where tumor growth is constrained by environmental limitations such as nutrient availability and spatial constraints. The inclusion of inter-compartment interactions further introduces transient fluctuations, reflecting the complex dynamics between metastatic sites.
The corresponding PSA observations, generated through a nonlinear observation model, closely follow the underlying tumor dynamics while exhibiting stochastic variability. This highlights an important clinical insight: PSA is an indirect and noisy surrogate marker of tumor burden. Although the overall trend of PSA aligns with tumor progression, the presence of measurement noise and nonlinear mapping implies that PSA alone may not reliably capture subtle changes in disease state. This observation is consistent with clinical findings that PSA-based monitoring can sometimes misrepresent true tumor activity.
The application of particle filtering provides a robust framework for addressing this challenge by enabling the estimation of latent tumor states from noisy observations. By approximating the posterior distribution of the tumor state using a set of weighted particles, the method effectively accounts for both model uncertainty and measurement noise. This is particularly advantageous in the context of prostate cancer, where tumor progression is governed by nonlinear biological processes and direct measurement of tumor burden is not feasible. The ability of particle filtering to provide both state estimates and uncertainty quantification enhances its potential utility for personalized disease monitoring and treatment planning.
Despite these strengths, several limitations should be acknowledged. The model relies on simplified assumptions regarding tumor growth, inter-site interactions, and PSA production. In reality, tumor biology is influenced by a wide range of factors including genetic heterogeneity, immune response, and treatment effects, which are not explicitly captured in the current framework. Additionally, parameter selection plays a critical role in shaping model behavior, and inaccurate parameterization may lead to misleading predictions. Future work should focus on incorporating patient-specific data, refining biological realism, and validating the model against clinical datasets.
In conclusion, the integration of nonlinear tumor dynamics, observation modeling, and particle filtering offers a promising approach for understanding prostate cancer progression and interpreting PSA measurements. This framework provides a foundation for developing more advanced predictive models that can support clinical decision making and improve patient outcomes.
5. Conclusion
In this study, a nonlinear state space framework was developed to model prostate cancer progression and its relationship to observable PSA dynamics. By integrating biologically motivated tumor growth equations with an observation model and particle filtering, the approach enables estimation of latent tumor burden from noisy PSA measurements. The results demonstrate that while PSA captures the general trend of disease progression, it remains an indirect and imperfect indicator of true tumor dynamics. The use of particle filtering provides a flexible and robust method for handling nonlinearities and uncertainties inherent in the system, offering both state estimates and uncertainty quantification. Overall, this framework highlights the potential of combining mathematical modeling and statistical inference to improve understanding of disease evolution and support more informed clinical decision making.