In ecosystems, the coexistence and evolution of species are influenced by multiple factors, including population migration, interspecific interactions, and environmental heterogeneity. In recent years, mathematical studies of predator-prey models have garnered increasing attention, where diffusion, chemotaxis, and fear effects are considered key mechanisms affecting the spatial distribution and dynamic behavior of populations.

In modeling predator-prey interactions, chemotaxis describes the directed movement response of organisms to external stimuli (such as food or predation risk). Kareiva and Odell [1] were among the first to propose a prey-taxis model to describe the aggregation behavior of predators towards areas of higher prey density. Subsequently, numerous studies have further explored predator-prey models incorporating prey-taxis [2]-[6], with reference [6] investigating prey-taxis in the Holling-Tanner predator-prey model. This study revealed the significant impact of the taxis coefficient as a bifurcation parameter on the existence of non-normal steady-state solutions and elucidated the crucial role of prey-taxis in the formation of spatial patterns.

In addition to the active search of predators for prey, prey also employ various anti-predation strategies to evade predation risk. Most species possess the ability to perceive predation risk and adjust their behavior accordingly [7]. Among these, predator-taxis is an important anti-predation behavior, where prey migrate towards areas of lower predator density in response to perceived predation risk. Previous research has proposed and analyzed predator-prey models that incorporate predator-taxis [8]-[12].

Moreover, fear effects are also significant factors influencing population dynamics. Wang et al. [13] were the first to introduce fear effects into a predator-prey model with a Holling-II functional response, finding that higher levels of fear could stabilize predator-prey dynamics by eliminating periodic solutions, while lower levels of fear might lead to multiple limit cycles through subcritical Hopf bifurcation, resulting in bistability. Subsequent studies [14]-[17] have further explored the impact of fear effects in various predator-prey models.

Inspired by the aforementioned studies, this paper considers a more complex predator-prey model. Unlike the research in reference [6], which focuses on predator foraging behavior (i.e., prey-taxis), this paper introduces both predator-taxis and fear effects into the Holling-Tanner predator-prey model. Additionally, the model incorporates environmental factors of spatial heterogeneity, where the intra-specific competition coefficient of predators, m ( x ) , depends on spatial location, thereby reflecting the complexity of real ecological environments. Building upon these motivations, we consider the full time-dependent model, which is a parabolic system describing the evolution of prey density u ( x , t ) and predator density v ( x , t )

This paper is devoted to the analysis of its steady-state problem, which yields the stationary solutions that dictate the long-term spatial patterns. Therefore, we investigate the following elliptic system

Let Ω be a bounded domain in ℝ n with a smooth boundary ∂ Ω , and let μ denote the unit outward normal vector on the boundary. The variables u and v represent the population densities of the prey and predator, respectively. The positive constants d 1 and d 2 are their random diffusion coefficients. The advection term − α ∇ ⋅ ( u ∇ v ) models predator-taxis, describing the directed movement of the prey away from regions of high predator density. The coefficient α > 0 represents the prey’s sensitivity to predation risk. The same risk perception triggers a fear effect, which reduces the prey’s reproduction rate; this is captured in the term r 0 u 1 + k 0 α v , where k 0 > 0 scales the impact of fear on the birth rate r 0 . The parameter d signifies the natural mortality rate of the prey, and a is the intra-specific competition coefficient among prey. The expression p u 1 + q u denotes the Holling-II functional response of the predator. The term s represents

the predator’s intrinsic growth rate, and − m ( x ) v 2 r + u indicates intra-specific competition among predators, where m ( x ) ≥ 0 is a spatially dependent coefficient and r > 0 is a half-saturation constant for the prey-dependent carrying capacity. The term c p u v 1 + q u represents biomass conversion, with c > 0 being the conversion efficiency. The homogeneous Neumann boundary conditions signify that the system is isolated at the boundary.

The novelty of this work with respect to the existing literature is twofold. While studies like [6] incorporate prey-taxis into the Holling-Tanner framework, and [8] -[12] analyze predator-taxis models, our work uniquely couples predator-taxis and fear effects through the same sensitivity coefficient α , which appears simultaneously in the cross-diffusion term − α ∇ ⋅ ( u ∇ v ) and the fear-modified growth rate r 0 u 1 + k 0 α v . This dual coupling creates a strongly nonlinear mathematical structure absent in all the aforementioned works. More importantly, unlike the constant-coefficient competition terms in [6][8]-[12], we introduce a spatially degenerate intra-specific competition coefficient m ( x ) for the predator. This degeneracy, characterized by m ( x ) ≡ 0 in a subregion Ω 0 , models a refuge scenario where predator competition completely vanishes locally. The vanishing of m ( x ) fundamentally alters the mathematical framework: it does not merely perturb a uniform state but gives rise to a new class of semi-trivial solutions and bifurcation structures, posing significant analytical challenges in establishing a priori bounds—a task that is central to our present analysis.

For the spatially dependent parameter m ( x ) , we assume it exhibits degeneracy within the region Ω, specifically that m ( x ) satisfies

where Ω ¯ 0 is a smooth closed subdomain within Ω. From a biological perspective, condition (3) indicates that Ω 0 is a suitable subregion for the survival of the predator population, as the intra-specific competition pressure among predators completely vanishes in the subregion Ω 0 .

The strong coupling characteristics of the model (2) primarily stem from two key factors: first, the system includes the cross-diffusion term α ∇ ⋅ ( u ∇ v ) ; and second, the taxis coefficient α appears simultaneously in the reaction term on the right-hand side of the equations. This dual association renders the model (2) a strongly coupled partial differential equation system, posing significant challenges for subsequent theoretical analysis.

The main objective of this paper is to investigate the uniform boundedness, local bifurcation structure, and local asymptotic stability of the positive steady-state solutions of the aforementioned model. Specifically, we first establish a priori estimates for the positive solutions, from which we prove the uniform boundedness of the positive steady-state solutions. Next, taking the growth rate of the prey population r 0 as a bifurcation parameter, we utilize the Crandall-Rabinowitz local bifurcation theorem to analyze the existence of positive solution branches bifurcating from the semi-trivial solution curve, and further discuss the linear stability of the branching solutions.

The primary contribution of this paper is a rigorous theoretical analysis, establishing uniform boundedness, existence via bifurcation, and local asymptotic stability for the positive steady-state solutions. We emphasize that the analytical thresholds derived herein—particularly the bifurcation parameter r 0 and the stability criteria—are not merely abstract mathematical conditions; ecologically, they delineate critical regimes for species coexistence. Specifically, when the prey growth rate r 0 crosses the bifurcation point, the system transitions from a predator-free state to a coexistence state, thereby predicting the precise conditions under which both species can persist in the heterogeneous landscape. The local stability or instability of the bifurcating solution translates directly into ecological forecasts: a stable positive steady state corresponds to the formation of a stationary spatial pattern of coexistence, whereas instability signals the onset of cyclical population dynamics or localized extinctions, especially near the boundary of the predator refuge Ω 0 where the degeneracy of m ( x ) takes effect. We acknowledge that this analytical work serves as a necessary theoretical foundation for subsequent numerical investigations. Without the a priori estimates and stability thresholds established here, any computational exploration of pattern formation would lack a solid mathematical basis, particularly given the degeneracy of the competition coefficient m ( x ) in the refuge region. The numerical visualization of these spatial patterns and the ecological implications of parameter variations will be addressed in our future work.

First, we define several symbols that will be used in the subsequent sections of this paper. Let T be a differential operator, ϕ be a Hölder continuous function on Ω, and d be a positive constant. Denote λ 1 D ( ± d , T + ϕ , Ω ) and λ 1 N ( ± d , T + ϕ , Ω ) as the first eigenvalues of the operator ∓ d Δ + T + ϕ in the domain Ω under Dirichlet and Neumann boundary conditions, respectively. When T = ϕ = 0 , we omit the operator T and the function ϕ , denoting λ 1 D ( ± d , Ω ) = λ 1 D ( ± d , 0 , Ω ) and λ 1 N ( ± d , Ω ) = λ 1 N ( ± d , 0 , Ω ) ; when only T = 0 , we omit the operator T , denoting λ 1 D ( ± d , ϕ , Ω ) = λ 1 D ( ± d , 0 + ϕ , Ω ) and λ 1 N ( ± d , ϕ , Ω ) = λ 1 N ( ± d , 0 + ϕ , Ω ) .

Furthermore, the norms in the spaces L p ( Ω ) ( p ≥ 1 ) and C ( Ω ¯ ) are defined as follows

According to Theorem 2.2 (i) in [18], when s < λ 1 D ( d 2 , Ω 0 ) , the boundary value problem

has a unique positive solution v s d 2 ( x ) ; when s ≥ λ 1 D ( d 2 , Ω 0 ) , there are no positive solutions to this problem, where the function m ( x ) satisfies condition (3). This indicates that when 0 < s < λ 1 D ( d 2 , Ω 0 ) , the model (2) has a semi-trivial solution ( 0 , v s d 2 ) . Additionally, when r 0 > d , the model (2) also has another semi-trivial solution ( r 0 − d a , 0 ) , and for any parameter values, the model always has the trivial solution ( 0 , 0 ) .

Next, we will prove that

is a necessary condition for the existence of positive solutions of the model (2), as stated in the following lemma.

Lemma 2.1The model(2)has positive solutions only if(5)holds.

Proof. We use proof by contradiction. Assume that when s ≥ λ 1 D ( d 2 , Ω 0 ) , the model (2) has a positive solution ( u , v ) , then v must be an upper solution of the boundary value problem (4). Let φ be the eigenfunction corresponding to the eigenvalue λ 1 D ( d 2 , Ω 0 ) . Extending φ to zero in the entire domain Ω, for sufficiently small ϵ > 0 , ϵ φ serves as a lower solution for the boundary value problem (4). By the method of upper and lower solutions for elliptic equations, there must exist a non-negative solution between the lower solution ϵ φ and the upper solution v , which is strictly positive in the interior of the domain Ω. This indicates that the boundary value problem (4) has a positive solution, contradicting the conclusion of Theorem 2.2 (i) in [18] (which states that when s ≥ λ 1 D ( d 2 , Ω 0 ) , the boundary value problem (4) has no positive solutions). Therefore, the assumption is false, and the original conclusion is proved.

From Lemma 2.1, we conclude that condition (5) is necessary for the existence of positive solutions of the model (2). Hence, in the subsequent content of this paper, we will always assume that condition (5) holds.

Theorem 2.2Let a , M , d be given positive constants satisfying the condition s < λ 1 D ( d 2 , Ω 0 ) . Then, when the parameter r 0 satisfies d < r 0 < a M + d , there exists a positive constant W = W ( M ) that depends only on M , such that anypositive solution ( u , v ) of the model(2)satisfies ‖ u ‖ ∞ + ‖ v ‖ ∞ ≤ W .

Proof. We employ a proof by contradiction. Assume that the conclusion does not hold. Then there exists a sequence { r 0 n } n = 1 ∞ such that d < r 0 n < a M + d , for which there exist positive solutions ( u n , v n ) of the model (2) satisfying

From the first equation of the model (2), u n satisfies

Noting that r 0 n u n 1 + k 0 α v n ≤ r 0 n u n and p u n v n 1 + q u n ≥ 0 , we obtain

Let U n be the positive solution of the ordinary differential equation r 0 n U − d U − a U 2 = 0 , which is given by U n = r 0 n − d a . By the maximum principle for elliptic equations, we find that

Thus, ‖ u n ‖ ∞ ≤ M , which indicates that { u n } is uniformly bounded. Combining this with equation (6), we conclude that ‖ v n ‖ ∞ → ∞ as n → ∞ .

Define v ^ n = v n ‖ v n ‖ ∞ . Dividing both sides of the second equation of the model (2) by ‖ v n ‖ ∞ , we find that v ^ n satisfies

Since m ( x ) ≥ 0 , v ^ n ≥ 0 , ‖ v n ‖ ∞ > 0 , and r + u n > 0 , it follows that

Moreover, since u n 1 + q u n < 1 q , we have

Multiplying both sides of this inequality by v ^ n and integrating over Ω using integration by parts yields

Adding ∫ Ω v ^ n 2 d x to both sides, and noting that ‖ v ^ n ‖ ∞ = 1 implies ∫ Ω v ^ n 2 d x ≤ | Ω | , we obtain

This shows that the sequence { v ^ n } n = 1 ∞ is uniformly bounded in H 1 ( Ω ) . Therefore, there exists a subsequence (still denoted as { v ^ n } ) and a function v ^ ∈ H 1 ( Ω ) such that

From ‖ v ^ n ‖ ∞ = 1 and the strong convergence, using the argument in Proposition 4.1 from [18], we can deduce that v ^ ≡ 0 holds in Ω ¯ , and it is evident that 0 ≤ v ^ ≤ 1 .

Next, we will prove that v ^ = 0 holds almost everywhere in Ω \ Ω ¯ 0 . Multiplying both sides of equation (7) by v ^ n and integrating over Ω gives us

Since { v ^ n } is bounded in H 1 ( Ω ) and ‖ u n ‖ ∞ ≤ M , the right-hand side of Equation (8) is uniformly bounded with respect to n . Given that ‖ v n ‖ ∞ → ∞ , it follows that the left-hand side must tend to zero

Due to the boundedness of { u n } , there exists a subsequence such that u n → u * weakly in L 2 ( Ω ) . Combining this with the strong convergence of v ^ n , we take the limit to obtain

Since m ( x ) > 0 in Ω \ Ω ¯ 0 and r + u * ≥ r > 0 , we conclude that v ^ = 0 holds almost everywhere in Ω \ Ω ¯ 0 .

Now consider the region Ω 0 . Since m ( x ) ≡ 0 on Ω ¯ 0 , Equation (7) simplifies in Ω 0 to

Multiplying this equation by a test function φ ∈ C 0 ∞ ( Ω 0 ) and integrating over Ω 0 , we obtain

Taking the limit as n → ∞ and using the weak convergence of v ^ n along with the boundedness of u n , we find that multiplying both sides by the test function φ ∈ C 0 ∞ ( Ω 0 ) yields

This indicates that v ^ | Ω 0 ∈ H 0 1 ( Ω 0 ) is a non-negative weak solution of the equation

Since v ^ ≡ 0 and v ^ = 0 in Ω \ Ω ¯ 0 , it must be that v ^ | Ω 0 ≡ 0 . By the Krein–Rutman theorem, s must be the principal eigenvalue of the operator − d 2 Δ − c p u * 1 + q u * on Ω 0 with homogeneous Dirichlet boundary conditions, i.e.,

Since u * ≥ 0 in Ω 0 , we have − c p u * 1 + q u * ≤ 0 . By the monotonicity of eigenvalues with respect to potential functions, we obtain

Thus, s ≥ λ 1 D ( d 2 , Ω 0 ) , which contradicts the assumption of the theorem that s < λ 1 D ( d 2 , Ω 0 ) . This contradiction indicates that the assumption made in the proof by contradiction is false, thereby proving the original statement.

In this subsection, we take r 0 as the bifurcation parameter and, using the local bifurcation theory proposed by Crandall and Rabinowitz [19], we study the local bifurcation structure of positive solutions of model (2) starting from the semi-trivial solutions ( r 0 − d a , 0 ) and ( 0 , v s d 2 ) , as well as the stability of these local bifurcation solutions.

Now, we investigate the local bifurcation phenomena of model (2). First, we introduce the function

where 0 < s < λ 1 D ( d 2 , Ω 0 ) and r 0 ∈ [ d , + ∞ ) . Since 1 1 + k 0 α v s d 2 > 0 , we know that τ ( r 0 ) is continuous and strictly increasing with respect to r 0 . Using a reasoning similar to Theorem 2.1 in [20] and combining standard regularity theory for elliptic equations, Sobolev embedding theorem, and the boundary smoothness assumption ∂ Ω ∈ C 2 + ϵ ( 0 < ϵ < 1 ) , we can show that there exists a constant M ∘ such that any positive solution ( u , v ) ∈ C 2 ( Ω ) × C 2 ( Ω ) of model (2) satisfies

Furthermore, assuming there exists a parameter d such that τ ( d ) < d , regardless of whether τ ( d ) < 0 , τ ( d ) = 0 , or τ ( d ) > 0 , there exists a unique r 0 ** ∈ [ d , + ∞ ) such that

Let p > N , define the function spaces X = W n 2 , p ( Ω ) × W n 2 , p ( Ω ) and Y = L p ( Ω ) × L p ( Ω ) , where

Model (2) has two semi-trivial solution curves Θ u and Θ v , which are given by

Next, we separately study the local bifurcation structure of positive solutions of model (2) from the semi-trivial solution curves Θ u and Θ v .

Theorem 3.1Let r 0 * = d − s a c p + s q , and r 0 * * be given by(9). The following conclusions hold

If − c p q < s < 0 , then the sufficient and necessary condition for the model (2) to produce a positive solution branch from the semi-trivial solution branch Θ u is r 0 = r 0 * . Furthermore, all positive solutions of model (2) in the neighborhood of the point ( r 0 * , r 0 * − d a , 0 ) ∈ ℝ × X lie on a smooth curve θ 1 , expressed as

where θ 1 > 0 is a constant, ϕ * and ψ * are determined functions in the space W n 2 , p ( Ω ) , and r ′ 0 ( 0 ) is defined as

Additionally, when p < p ˜ , the solution branch θ 1 at the point ( r 0 * , r 0 * − d a , 0 ) undergoes a supercritical bifurcation; when p > p ˜ , this bifurcation is subcritical, where the critical parameter p ˜ satisfies

If 0 < s < λ 1 D ( d 2 , Ω 0 ) , then the sufficient and necessary condition for the model (2) to produce a positive solution branch from the semi-trivial solution branch Θ v is r 0 = r 0 ** . Furthermore, all positive solutions of model (2) in the neighborhood of the point ( r 0 ** , 0 , v s d 2 ) ∈ ℝ × X lie on a smooth curve θ 2 , expressed as

where θ 2 > 0 is a constant, ϕ ** and ψ ** are determined functions in the space W n 2 , p ( Ω ) , and r ^ ′ 0 ( 0 ) is defined as

Additionally, when a > a ˜ , the solution branch θ 2 at the point ( r 0 ** , 0 , v s d 2 ) undergoes a supercritical bifurcation; when a < a ˜ , this bifurcation is subcritical, where the critical parameter a ˜ satisfies

Proof. The proof of conclusion (i) is completely similar to that of (ii), so we will only provide a detailed proof of conclusion (ii) here. Let w = v − v s d 2 , and define the nonlinear operator L 1 : ℝ × X → Y as

where the component expressions are given by

Clearly, the sufficient and necessary condition for the equation L 1 ( r 0 , u , w ) = 0 to hold is that ( u , w + v s d 2 ) is a solution of model (2). Therefore, to find the positive solution branch of (2) near ( r 0 , u , v ) = ( r 0 , 0 , v s d 2 ) , we only need to find the positive solution branch of L 1 ( r 0 , u , w ) = 0 near ( r 0 , u , w ) = ( r 0 , 0 , 0 ) .

Let L 1 ( u , w ) ( r 0 , u , w ) denote the Fréchet derivative of the operator L 1 with respect to the variables ( u , w ) , and direct calculation gives

where the coefficient operators B 1 , B 2 , B 3 are defined as

Based on the Krein-Rutman theorem, the existence of a positive solution ( ϕ , ψ ) satisfying ϕ > 0 for the equation L 1 ( u , w ) ( r 0 , 0 , 0 ) ( ϕ , ψ ) ⊤ = ( 0 , 0 ) ⊤ is equivalent to the condition r 0 = r 0 ** . Furthermore, it can be verified that the point ( r 0 ** , 0 , v s d 2 ) is the unique bifurcation point from the semi-trivial solution branch Θ v of model (2) producing a positive solution branch.

Let ( ϕ * * , ψ * * ) ∈ ker L 1 ( u , w ) ( r 0 * * , 0 , 0 ) , then ( ϕ ** , ψ ** ) satisfies

First, we analyze the first equation of (10). Based on this equation, we construct the following Neumann boundary value eigenvalue problem

According to the previously defined notation, the principal eigenvalue of the above eigenvalue problem can be denoted as

where the parameters satisfy 0 < s < λ 1 D ( d 2 , Ω 0 ) and r 0 ∈ ( d , + ∞ ) . From the monotonicity of the principal eigenvalue of the elliptic operator, we know that the principal eigenvalue strictly increases with the potential function. Since the coefficient of r 0 , 1 1 + k 0 α v s d 2 > 0 , it follows that ξ ( r 0 ) is strictly increasing with respect to r 0 , and as r 0 → + ∞ , ξ ( r 0 ) → + ∞ . Combining ξ ( r 0 ** ) = d , we conclude that the first equation of (10) has a unique positive solution ϕ * * ∈ W n 2 , p ( Ω ) if and only if r 0 = r 0 ** . Next, we analyze the second equation of (10). From equation (4), we know that

which indicates that λ 1 N ( d 2 Δ + s − m ( x ) v s d 2 r , Ω ) = 0 . Again using the monotonicity of the principal eigenvalue, since the potential function satisfies − 2 m ( x ) v s d 2 r < − m ( x ) v s d 2 r , we have

For the elliptic operator Δ, its principal eigenvalue is the unique maximum eigenvalue among all eigenvalues. Therefore, all eigenvalues of the operator d 2 Δ + s − 2 m ( x ) v s d 2 r are less than 0, indicating that this operator is strictly negative, and consequently, its inverse ( d 2 Δ + s − 2 m ( x ) v s d 2 r ) − 1 exists and is strictly negative. Thus, the second equation of (10) can be equivalently transformed into

Since ϕ ** > 0 and m ( x ) ( v s d 2 ) 2 r 2 + c p v s d 2 > 0 , combined with the negativity of the inverse operator, we conclude that ψ ** > 0 . In summary, we can prove that ker ( L 1 ( u , w ) ( r 0 ** , 0 , 0 ) ) = Span { ( ϕ ** , ψ ** ) } , indicating that the dimension of this kernel space is

Next, we verify that codim ( Range ( L 1 ( u , w ) ( r 0 ** , 0 , 0 ) ) ) = 1 . Consider the adjoint operator L 1 ( u , w ) ** ( r 0 ** , 0 , 0 ) . According to functional analysis theory, the kernel space of the adjoint operator is orthogonal to the range of the original operator, i.e., Range ( L 1 ( u , w ) ( r 0 ** , 0 , 0 ) ) ⊥ = ker ( L 1 ( u , w ) ** ( r 0 ** , 0 , 0 ) ) . The matrix form of the operator L 1 ( u , w ) ** ( r 0 ** , 0 , 0 ) can be expressed as

where

To determine the structure of ker ( L 1 ( u , w ) ** ( r 0 ** , 0 , 0 ) ) , let ( ϕ ′ ** , ψ ′ ** ) ⊤ ∈ ker ( L 1 ( u , w ) ** ( r 0 ** , 0 , 0 ) ) , then it satisfies

Expanding gives the following Neumann boundary value problem