In this paper, we consider the Cauchy problem for the following nonlinear Schrödinger type equations with mixed power-type and Hartree-type nonlinearities.

$\{\begin{array}{l}i{\partial }_t\psi +\text{Δ}\psi +{\left|\psi \right|}^p\psi +\left({\left|x\right|}^{-\gamma }\ast {\left|\psi \right|}^2\right)\psi =0,\left(t,x\right)\in \left[0,T\right)\times {ℝ}^N,\hfill \\ \psi \left(0,x\right)={\psi }_0\left(x\right)\in H^1\left({ℝ}^N\right).\hfill \end{array}$(1)

where $\psi =\psi \left(t,x\right)$ is a complex valued function of $\left(t,x\right)\in {ℝ}^{+}\times {ℝ}^N$. The parameters $0<\gamma <\min \left\{4,N\right\}$, $N\ge 3$ and $0<p<\frac{4}{N}$. Equation (1) has several

physical origins and backgrounds, which appeared in many physical models and depended on different parameter configuration. Hartree equation was proposed by Hartree in 1928 to describe the multi-electron wave system. He believed that the multi-electron wave function in the system can be regarded as the product of multiple single-electron wave functions, which can be obtained by calculating the energy minimum by variational method. After improvements by Fock et al., the theory was called Hartree-Fock theory. Since its introduction, Hartree-Fock theory has become a basic tool in many fields, such as quantum physics, chemistry and plasma physics. The Hartree equation is closely related to the nonlinear Schrodinger equation, which is considered to be the Hartree equation in the mean field model. From the intrinsic structure, the Hartree equation can reveal the interaction between electrons more deeply in some specific situations, which makes it unique in dealing with multi-electron systems.

In recent years, this type of equation has been studied extensively in [1] - [12] . Pekar describes the quantum theory of resting polarons in mathematical physics [13] . For the Hartree-type nonlinearity $\left({\left|x\right|}^{-\gamma }\ast {\left|\psi \right|}^2\right)\psi$, Cho et al. proved the existence and uniqueness of local and global solutions of Equation (1) in [14] , and they also showed the existence of blow-up solutions in [15] . Feng [16] and Zhu [12] studied the orbital stability of the standing wave solution of the fractional Schrodinger equation.

In general, Equation (1) admits a class of special solutions, which are called standing waves, namely the time-periodic solitary waves of the form $\psi \left(t,x\right)={\text{e}}^{i\omega t}u\left(x\right)$, where $\omega \in ℝ$ is a frequency and $u\in H^1\left({ℝ}^N\right)$ is a nontrivial solution satisfies the elliptic equation

$-\text{Δ}u+\omega u={\left|u\right|}^{p}u+\left({\displaystyle {\int }_{{ℝ}^N}\frac{{\left|u\right|}^2}{{\left|x-y\right|}^{\gamma }}\text{d}y}\right)u.$(2)

When considering Equation (2), there are two possible methods in terms of the frequency $\omega$. The first one is to fix the frequency $\omega \in ℝ$, and the other can obtain the existence of solutions by studying the critical point of the action functional $A_{\omega }\left(u\right)$ on $H^1\left({ℝ}^N\right)$, where

$A_{\omega }\left(u\right):=\frac{1}{2}{\Vert \nabla u\Vert }_{L^2}^2-\frac{1}{p+2}{\Vert u\Vert }_{L^{p+2}}^{p+2}-\frac{1}{4}{\displaystyle {\int }_{{ℝ}^N}{\displaystyle {\int }_{{ℝ}^N}\frac{{\left|u\right|}^2{\left|u\right|}^2}{{\left|x-y\right|}^{\gamma }}\text{d}x\text{d}y}}+\frac{\omega }{2}{\Vert u\Vert }_{L^2}^2.$(3)

In this case, we are mainly concerned with the existence of minimal action solutions, namely solutions minimizing $A_{\omega }\left(u\right)$ among all nontrivial solutions.

On the other hand, it is interesting to study solutions for Equation (2), having prescribed $L^2$-norm. That is, for any given $c>0$, consider solutions of Equation (2) with the $L^2$-norm constrain

$S\left(c\right):=\left\{u\in H^1,{\Vert u\Vert }_{L^2}^2=c\right\}.$

Physically, the associated energy function is defined on $H^1\left({ℝ}^N,C\right)$ by

$E\left(u\right)=\frac{1}{2}{\Vert \nabla u\Vert }_{L^2}^2-\frac{1}{p+2}{\Vert u\Vert }_{L^{p+2}}^{p+2}-\frac{1}{4}{\displaystyle {\int }_{{ℝ}^N}{\displaystyle {\int }_{{ℝ}^N}\frac{{\left|u\right|}^2{\left|u\right|}^2}{{\left|x-y\right|}^{\gamma }}\text{d}x\text{d}y}}.$(4)

In particular, the frequency $\omega \in ℝ$ is determined as the Lagrange multiplier associated with the $S\left(c\right)$.

Next, we shall focus on the existence of a ground state and recall this definition.

Definition 1.1. We say that $u_c$ is a ground state of Equation (2) on $S\left(c\right)$ if it is a solution having minimal energy among all the solutions which belong to $S\left(c\right)$. Namely, if

$E\left(u_c\right)=\inf \left\{E\left(u\right),u\in S\left(c\right),{\left({E|}_{S\left(c\right)}\right)}^{\prime }\left(u\right)=0\right\}.$

Since the $L^2$-norm is a preserved quantity of the evolution, the variational characterization of these solutions is often extremely useful in analyzing their orbital stability, see [9] [17] - [19] . Therefore, we decide to study the orbital stability of standing wave solutions by variational properties.

In particular, for any $c\in \left(0,c_0\right)$, there exists a set $V\left(c\right)\subset S\left(c\right)$ having the property that

$m\left(c\right):=\underset{u\in V\left(c\right)}{\text{inf}}E\left(u\right)<0<\underset{u\in \partial V\left(C\right)}{\text{inf}}E\left(u\right).$

The set $V\left(c\right)$ and $\partial V\left(c\right)$ are give by

$V\left(c\right):=\left\{u\in S\left(c\right):{\Vert \nabla u\Vert }_{L^2}^2<{\rho }_0\right\},\partial V\left(c\right):=\left\{u\in S\left(c\right):{\Vert \nabla u\Vert }_{L^2}^2={\rho }_0\right\},$(5)

for a ${\rho }_0>0$ that depends only on $c_0>0$, we define the following set

${ℳ}_c:=\left\{u\in V\left(c\right):E\left(u\right)=m\left(c\right)\right\}.$

Theorem 1.2. For any $c\in \left(0,c_0\right)$, if $\left(u_n\right)\subset B_{{\rho }_0}$ is such that ${\Vert u_n\Vert }_{L^2}^2\to c$ and $E\left(u_n\right)\to m\left(c\right)$. Then, up to translation, $u_n\stackrel{H}{\to }u\in {ℳ}_c$. In particular, the set ${ℳ}_c$ is compact in H, and it is orbitally stable.

Orbital stability emphasizes the relative relationship between orbits, and focuses on whether the relative position of the trajectory of the solution remains stable. It is not required that the orbit tends to a particular state over time. But other stability, such as asymptotic stability requires both the proximity of the initial moment and the eventual tendency towards a particular solution over time.

Generally speaking, there are two main methods to study the orbital stability of the standing wave solution of the Schrodinger equation. One is to discuss the spectral properties of the operator ${L^{″}}_{\omega }\left(Q_{\omega }\right)$ and to judge the sign of ${\partial }_{\omega }{\Vert \Vert }_{L^2}^2$, where $L_{\omega }\left(Q_{\omega }\right)$ is the corresponding Lyapunov functional. This is the method for judging the stability of standing wave solutions given by Shatah, Strauss and Grillakis [20] [21] . This discriminant method is very useful for studying the stability and instability of the standing wave solution of the Schrodinger equation with the homogeneous nonlinear term. However, for the Schrodinger equation with mixed power nonlinear terms such as Equation (1), The calculation of ${\partial }_{\omega }{\Vert \Vert }_{L^2}^2$ is very difficult. Another method is the research framework for the stability of standing wave solutions established by Cazenave and Lions [18] , that is, to prove the stability of standing wave solutions by studying the minimization problem under mass constraint. This method uses only two properties: the conservation law of the Schrodinger equation and the compactness of any minimization sequence. And compactness needs to be obtained by proving the concentration compactness principle. Therefore, the research method of Cazenave-Lions has been widely used in the study of the stability of the standing wave solution of the Schrodinger equation, see references [18] [22] [23] .

The orbital stability of the standing wave solution is studied mainly by solving the variational problem with fixed mass to obtain the ground state solution, and then using the law of conservation of mass and conservation of energy, the orbital stability of the standing wave solution is established, see [5] [10] .

In physics, the ground state usually refers to the state in which a quantum system reaches its minimal energy. Mathematically, the ground state solution of Equation (2) is the one that gives Equation (1) the minimal energy among all the solutions of Equation (2). It is generally obtained by solving the variational problem of Equation (2), and the corresponding standing wave solution will generally have some stability. The definition of orbital stability of Equation (1) is given below.

Definition 1.3. We say that the set ${ℳ}_c$ is orbitally stable. Let $u\in S\left(c\right)$ is a ground state solution to Equation (2) if for each $\epsilon >0$, there is a $\delta >0$ such that

if initial data ${\psi }_0\in H^1\left({ℝ}^N\right)$ and $\underset{u\in {ℳ}_c}{\inf }{\Vert {\psi }_0-u\Vert }_{H^1\left({ℝ}^N\right)}<\delta$, then the corresponding solution to Equation (1) with ${\psi |}_{t=0}={\psi }_0$ satisfies

$\underset{t\in ℝ}{\text{sup}}\underset{u\in {ℳ}_c}{\text{inf}}{\Vert \psi \left(t,\cdot \right)-u\Vert }_{H^1\left({ℝ}^N\right)}<\epsilon ,$

then the set ${ℳ}_c$ of energy minimizers is said to be orbitally stable. Otherwise, it is unstable.

In this section, we recall some preliminary results that will be used later. Firstly, let us recall the local theory for the Cauchy problem Equation (1) established in [14] [24] .

Lemma 2.1. Let $0<p<\frac{4}{N-2}$, define Weinstein’s functional

$\alpha :=\underset{0\ne u\in H^1}{\text{inf}}{I}^{N,p}\left(u\right)=\underset{0\ne u\in H^1}{\text{inf}}\frac{{\left({\displaystyle \int {\left|u\right|}^2\text{d}x}\right)}^{\frac{4-\left(N-2\right)p}{4}}{\left({\displaystyle \int {\left|\nabla u\right|}^2\text{d}x}\right)}^{\frac{Np}{4}}}{{\displaystyle \int {\left|u\right|}^{p+2}\text{d}x}}.$

Then, $I^{N,p}\left(u\right)$ is attained at a function R, satisfying

(i) $R:=R\left(\left|x\right|\right)$ is positive in $H^1\cap C^{\infty }$;

(ii) R is a solution of the following equation of the minimal $L^2$-norm (ground state)

$-\Delta R+\frac{4-\left(N-2\right)p}{Np}R-{\left|R\right|}^{p}R=0,R\in H^1;$(6)

(iii) $\alpha =\frac{Np}{2\left(p+2\right)}{\Vert \nabla R\Vert }_{L^2}^p$ Therefore, by the proposition above, we can obtain the following sharp Gagliardo-Nirenberg inequality: for all $0<p<\frac{4}{N-2}$, and $u\in H^1\left({ℝ}^N\right)$

${\Vert u\Vert }_{L^{p+2}}^{p+2}\le \frac{2\left(p+2\right)}{Np{\Vert \nabla R\Vert }_{L^2}^p}{\Vert u\Vert }_{L^2}^{p+2-\frac{Np}{2}}{\Vert \nabla u\Vert }_{L^2}^{\frac{Np}{2}}.$(7)

where R is the unique positive ground state solution of Equation (2). For the Hartree nonlinearity, according to the results in [25] - [27] , and we consider the following elliptic equation:

$-\Delta W+\frac{4-\gamma }{\gamma }W-\left({\displaystyle {\int }_{{ℝ}^N}\frac{{\left|W\right|}^2}{{\left|x-y\right|}^{\gamma }}\text{d}x\text{d}y}\right)W=0,W\in H^1.$

In the paper, we also need the following sharp Gagliardo-Nirenberg inequality, see [25] [27]

Lemma 2.2. Let $0<\gamma <\min \left\{4,N\right\}$. For all $u\in H^1$, we have

${\displaystyle {\int }_{{ℝ}^N}{\displaystyle {\int }_{{ℝ}^N}\frac{{\left|u\left(x\right)\right|}^2{\left|u\left(y\right)\right|}^2}{{\left|x-y\right|}^{\gamma }}\text{d}x\text{d}y}}\le \frac{4}{\gamma {\Vert \nabla W\Vert }_{L^2}^2}{\Vert u\Vert }_{L^2}^{4-\gamma }{\Vert \nabla u\Vert }_{L^2}^{\gamma },$(8)

where W is a positive ground state solution of Equation (6).

Lemma 2.3. [28] Let $N\ge 3$ and ${\left\{u_n\right\}}_{n=1}^{\infty }$ be a bounded sequence in $H^1\left({ℝ}^N\right)$ satisfying:

${\displaystyle {\int }_{{ℝ}^N}{\left|u_n\right|}^2\text{d}x}=\lambda ,$

where $\lambda >0$ is fixed. Then, there exists a subsequence ${\left\{u_{n_k}\right\}}_{k=1}^{\infty }$ satisfying one of the three possibilities:

(i) (compactness) there exists ${\left\{y_{n_k}\right\}}_{k=1}^{\infty }\subset {ℝ}^N$ such that $u_{n_k}\left(\cdot -y_{n_k}\right)\to u$ as $k\to \infty$ in $L^2\left({ℝ}^N\right)$, namely

$\forall \epsilon >0,\exists R>0,{\displaystyle {\int }_{B_R\left(y_{n_k}\right)}{\left|u_{n_k}\left(x\right)\right|}^2\text{d}x}\ge \lambda -\epsilon ;$

(ii) (vanishing)

$\underset{k\to \infty }{\text{lim}}\underset{y\in {ℝ}^N}{\text{sup}}{\displaystyle {\int }_{B_R\left(y\right)}{\left|u_{n_k}\left(x\right)\right|}^2\text{d}x}=0$ for all $R<\infty$;

(iii) (dichotomy) there exists $\sigma \in \left(0,\lambda \right)$ and $u_{n_k}^{\left(1\right)}$, $u_{n_k}^{\left(2\right)}$ bounded in $H^1\left({ℝ}^N\right)$ such that:

$\{\begin{array}{l}\left|u_{n_k}^{\left(1\right)}\right|+\left|u_{n_k}^{\left(2\right)}\right|\le \left|u_{n_k}\right|,\\ Supp{u}_{n_k}^{\left(1\right)}\cap Supp{u}_{n_k}^{\left(2\right)}=\varnothing ,\\ {\Vert u_{n_k}^{\left(1\right)}\Vert }_{H^1}+{\Vert u_{n_k}^{\left(2\right)}\Vert }_{H^1}\le C{\Vert u_{n_k}\Vert }_{H^1},\\ {\Vert u_{n_k}^{\left(1\right)}\Vert }_{L^2}^2\to \sigma ,{\Vert u_{n_k}^{\left(2\right)}\Vert }_{L^2}^2\to \lambda -\sigma ,\text{as}k\to \infty ,\\ \underset{k\to \infty }{\lim \inf }{\displaystyle {\int }_{{ℝ}^N}\left({\left|\nabla u_{n_k}\right|}^2-{\left|\nabla u_{n_k}^{\left(1\right)}\right|}^2-{\left|\nabla u_{n_k}^{\left(2\right)}\right|}^2\right)\text{d}x}\ge 0,\\ {\Vert u_{n_k}-\left(u_{n_k}^{\left(1\right)}+u_{n_k}^{\left(2\right)}\right)\Vert }_{L^s}\to 0\text{as}k\to 0\text{for}\text{all}2\le s<\frac{2N}{N-2}\left(2\le s<\infty \text{if}N=1\right).\end{array}$ (9)

In this section, we mainly consider the $L^2$-subcritical case and prove Theorem 1.2.

Lemma 3.1. For any $u\in S\left(c\right)$, we have that

$E\left(u\right)\ge {\Vert \nabla u\Vert }_{L^2}^{2}f\left(c,{\Vert \nabla u\Vert }_{L^2}^2\right).$

Proof. Let’s use Lemma 2.1 and Lemma 2.2; for any $u\in S\left(c\right)$, we have that,

$\begin{array}{c}E\left(u\right)=\frac{1}{2}{\Vert \nabla u\Vert }_{L^2}^2-\frac{1}{p+2}{\Vert u\Vert }_{L^p+2}^{p+2}-\frac{1}{4}{\displaystyle {\int }_{{ℝ}^N}{\displaystyle {\int }_{{ℝ}^N}\frac{{\left|u\right|}^2{\left|u\right|}^2}{{\left|x-y\right|}^{\gamma }}\text{d}x\text{d}y}}\\ \ge \frac{1}{2}{\Vert \nabla u\Vert }_{L^2}^2-\frac{1}{p+2}\frac{2\left(p+2\right)}{Np{\Vert \nabla R\Vert }_{L^2}^p}{\Vert \nabla u\Vert }_{L^2}^{\frac{Np}{2}}{\Vert u\Vert }_{L^2}^{p+2-\frac{Np}{2}}-\frac{1}{4}\frac{4}{\gamma {\Vert \nabla W\Vert }_{L^2}^2}{\Vert \nabla u\Vert }_{L^2}^{\gamma }{\Vert u\Vert }_{L^2}^{4-\gamma }\\ ={\Vert \nabla u\Vert }_{L^2}^2\left[\frac{1}{2}-\frac{2}{Np{\Vert \nabla R\Vert }_{L^2}^p}{\Vert \nabla u\Vert }_{L^2}^{\frac{Np}{2}-2}{\Vert u\Vert }_{L^2}^{p+2-\frac{Np}{2}}-\frac{1}{\gamma {\Vert \nabla W\Vert }_{L^2}^2}{\Vert \nabla u\Vert }_{L^2}^{\gamma -2}{\Vert u\Vert }_{L^2}^{4-\gamma }\right]\\ ={\Vert \nabla u\Vert }_{L^2}^{2}f\left({\Vert u\Vert }_{L^2}^2,{\Vert \nabla u\Vert }_{L^2}^2\right)\\ ={\Vert \nabla u\Vert }_{L^2}^{2}f\left(c,{\Vert \nabla u\Vert }_{L^2}^2\right).\end{array}$

Now, letting

${\alpha }_0=\frac{Np}{2}-2,{\alpha }_1=p+2-\frac{Np}{2},{\alpha }_2=\gamma -2,{\alpha }_3=4-\gamma ,$

we consider the function $f\left(c,\rho \right)$ defined on $\left(0,\infty \right)\times \left(0,\infty \right)$ by

$f\left(c,\rho \right)=\frac{1}{2}-\frac{2}{Np{\Vert \nabla R\Vert }_{L^2}^p}{\rho }^{\frac{{\alpha }_0}{2}}{c}^{\frac{{\alpha }_1}{2}}-\frac{1}{\gamma {\Vert \nabla W\Vert }_{L^2}^2}{\rho }^{\frac{{\alpha }_2}{2}}{c}^{\frac{{\alpha }_3}{2}},$(10)

and for each $c\in \left(0,\infty \right)$, its restriction $g_c\left(\rho \right)$ defined on $\left(0,\infty \right)$ by $\rho \mapsto g_c\left(\rho \right):=f\left(c,\rho \right)$. A reference to the following text, note that for any $N\ge 3$, $0<\gamma <\min \left\{4,N\right\}$, there are two cases

(1) $3\le N\le 4$

${\alpha }_0\in \left(-2,0\right),{\alpha }_1\in \left(\frac{4}{N},2\right),{\alpha }_2\in \left(0,2\right),{\alpha }_3\in \left(0,2\right),$

(2) $N\ge 5$

${\alpha }_0\in \left(-2,0\right),{\alpha }_1\in \left(\frac{4}{N},2\right),{\alpha }_2\in \left(0,N-2\right),{\alpha }_3\in \left(4-N,2\right).$

Lemma 3.2. For each $c>0$, the function $g_c\left(\rho \right)$ has a unique global maximum and the maximum value satisfies

$\{\begin{array}{l}\underset{\rho >0}{\text{max}}{g}_c\left(\rho \right)>0\text{if}c<c_0,\hfill \\ \underset{\rho >0}{\text{max}}{g}_c\left(\rho \right)=0\text{if}c=c_0,\hfill \\ \underset{\rho >0}{\text{max}}{g}_c\left(\rho \right)<0\text{if}c>c_0,\hfill \end{array}$

where

$c_0=\frac{1}{2\left(\mathcal{A}+ℬ\right)}{)}^{\frac{{\alpha }_1{\alpha }_2-{\alpha }_0{\alpha }_3}{2\left({\alpha }_2-{\alpha }_0\right)}}>0,$(11)

with

$\begin{array}{l}\mathcal{A}=\frac{2}{Np{\Vert \nabla R\Vert }_{L^2}^p}{\left(-\frac{2{\alpha }_0}{{\alpha }_2}\frac{\gamma {\Vert \nabla W\Vert }_{L^2}^2}{Np{\Vert \nabla R\Vert }_{L^2}^p}\right)}^{\frac{{\alpha }_0}{{\alpha }_2-{\alpha }_0}}>0,\\ ℬ=\frac{1}{\gamma {\Vert \nabla W\Vert }_{L^2}^2}{\left(-\frac{2{\alpha }_0}{{\alpha }_2}\frac{\gamma {\Vert \nabla W\Vert }_{L^2}^2}{Np{\Vert \nabla R\Vert }_{L^2}^p}\right)}^{\frac{{\alpha }_2}{{\alpha }_2-{\alpha }_0}}>0.\end{array}$

Proof. By the definition of $g_c\left(\rho \right)$, we can obtain by some calculation that

${g^{\prime }}_c\left(\rho \right)=-\frac{{\alpha }_0}{Np{\Vert \nabla R\Vert }_{L^2}^p}{\rho }^{\frac{{\alpha }_0}{2}-1}{c}^{\frac{{\alpha }_1}{2}}-\frac{{\alpha }_2}{\gamma {\Vert \nabla W\Vert }_{L^2}^2}{\rho }^{\frac{{\alpha }_1}{2}-1}{c}^{\frac{{\alpha }_3}{2}}.$

Therefore, there is unique solution of the equation $g_c^{\text{'}}\left(\rho \right)=0$, namely

${\rho }_c={\left(-\frac{2{\alpha }_0}{{\alpha }_2}\frac{\gamma {\Vert \nabla W\Vert }_{L^2}^2}{Np{\Vert \nabla R\Vert }_{L^2}^p}\right)}^{\frac{2}{{\alpha }_2-{\alpha }_0}}{c}^{\frac{{\alpha }_1-{\alpha }_3}{{\alpha }_2-{\alpha }_0}}.$(12)

Taking into account that $g_c\left(\rho \right)\to -\infty$ as $\rho \to 0$ and $g_c\left(\rho \right)\to -\infty$ as $\rho \to +\infty$. Therefore, we obtain that ${\rho }_c$ is the unique global maximum point of $g_c\left(\rho \right)$ and the maximum value is

$\begin{array}{l}\underset{\rho >0}{\text{max}}{g}_c\left(\rho \right)\\ =\frac{1}{2}-\frac{2}{Np{\Vert \nabla R\Vert }_{L^2}^p}{\left[{\left(-\frac{2{\alpha }_0}{{\alpha }_2}\frac{\gamma {\Vert \nabla W\Vert }_{L^2}^2}{Np{\Vert \nabla R\Vert }_{L^2}^p}\right)}^{\frac{2}{{\alpha }_2-{\alpha }_0}}{c}^{\frac{{\alpha }_1-{\alpha }_3}{{\alpha }_2-{\alpha }_0}}\right]}^{\frac{{\alpha }_0}{2}}{c}^{\frac{{\alpha }_1}{2}}\\ -\frac{1}{\gamma {\Vert \nabla W\Vert }_{L^2}^2}{\left[{\left(-\frac{2{\alpha }_0}{{\alpha }_2}\frac{\gamma {\Vert \nabla W\Vert }_{L^2}^2}{Np{\Vert \nabla R\Vert }_{L^2}^p}\right)}^{\frac{2}{{\alpha }_2-{\alpha }_0}}{c}^{\frac{{\alpha }_1-{\alpha }_3}{{\alpha }_2-{\alpha }_0}}\right]}^{\frac{{\alpha }_2}{2}}{c}^{\frac{{\alpha }_3}{2}}\\ =\frac{1}{2}-\left[\frac{2}{Np{\Vert \nabla R\Vert }_{L^2}^p}{\left(-\frac{2{\alpha }_0}{{\alpha }_2}\frac{\gamma {\Vert \nabla W\Vert }_{L^2}^2}{Np{\Vert \nabla R\Vert }_{L^2}^p}\right)}^{\frac{{\alpha }_0}{{\alpha }_2-{\alpha }_0}}+\frac{1}{\gamma {\Vert \nabla W\Vert }_{L^2}^2}{\left(-\frac{2{\alpha }_0}{{\alpha }_2}\frac{\gamma {\Vert \nabla W\Vert }_{L^2}^2}{Np{\Vert \nabla R\Vert }_{L^2}^p}\right)}^{\frac{{\alpha }_2}{{\alpha }_2-{\alpha }_0}}\right]c^{\tau }\\ =\frac{1}{2}-\left(\mathcal{A}+ℬ\right)c^{\tau }.\end{array}$

where, $\tau =\frac{{\alpha }_1{\alpha }_2-{\alpha }_0{\alpha }_3}{2\left({\alpha }_2-{\alpha }_0\right)}$. In view of Equation (11), we can obtain that $\underset{\rho >0}{\text{max}}{g}_{c_0}\left(\rho \right)=0$. The lemma is proved.

Lemma 3.3. Let $\left(c_1,{\rho }_1\right)\in \left(0,\infty \right)\times \left(0,\infty \right)$ be such that $f\left(c_1,{\rho }_1\right)\ge 0$. Then, for any $c_2\in \left(0,c_1\right]$, we can obtain that

$f\left(c_1,{\rho }_1\right)\ge 0$ if ${\rho }_2\in \left[\frac{c_2}{c_1}{\rho }_1,{\rho }_1\right]$.

Proof. It is shown that $c\mapsto f\left(\cdot ,\rho \right)$ is a non-increasing function, then

$f\left(c_2,{\rho }_1\right)\ge f\left(c_1,{\rho }_1\right)\ge 0.$(13)

Now, by some basic calculations, ${\alpha }_0+{\alpha }_1=p>0$ and taking into account

$f\left(c_1,{\rho }_1\right)=\frac{1}{2}-\frac{2}{Np{\Vert \nabla R\Vert }_{L^2}^p}{\rho }_1^{\frac{{\alpha }_0}{2}}{c}_1^{\frac{{\alpha }_1}{2}}-\frac{1}{\gamma {\Vert \nabla W\Vert }_{L^2}^2}{\rho }_1^{\frac{{\alpha }_2}{2}}{c}_1^{\frac{{\alpha }_3}{2}},$

and

$\begin{array}{l}f\left(c_2,\frac{c_2}{c_1}{\rho }_1\right)=\frac{1}{2}-\frac{2}{Np{\Vert \nabla R\Vert }_{L^2}^p}{\rho }_1^{\frac{{\alpha }_0}{2}}{c}_2^{\frac{{\alpha }_0+{\alpha }_1}{2}}{c}_1^{-\frac{{\alpha }_0}{2}}-\frac{1}{\gamma {\Vert \nabla W\Vert }_{L^2}^2}{\rho }_1^{\frac{{\alpha }_2}{2}}{c}_2^{\frac{{\alpha }_2+{\alpha }_3}{2}}{c}_1^{\frac{-{\alpha }_2}{2}}\hfill \end{array}$

Therefore

$\begin{array}{l}f\left(c_2,\frac{c_2}{c_1}{\rho }_1\right)-f\left(c_1,{\rho }_1\right)\\ =\frac{2}{Np{\Vert \nabla R\Vert }_{L^2}^p}{\rho }_1^{\frac{{\alpha }_0}{2}}\left(c_1^{\frac{{\alpha }_1}{2}}-c_2^{\frac{{\alpha }_0+{\alpha }_1}{2}}{c}_1^{-\frac{{\alpha }_0}{2}}\right)+\frac{1}{\gamma {\Vert \nabla W\Vert }_{L^2}^2}{\rho }_1^{\frac{{\alpha }_2}{2}}\left(c_1^{\frac{{\alpha }_3}{2}}-c_2^{\frac{{\alpha }_2+{\alpha }_3}{2}}{c}_1^{-\frac{{\alpha }_2}{2}}\right)\\ \ge \frac{2}{Np{\Vert \nabla R\Vert }_{L^2}^p}{\rho }_1^{\frac{{\alpha }_0}{2}}\left(c_1^{\frac{{\alpha }_1}{2}}-c_1^{\frac{{\alpha }_0+{\alpha }_1}{2}}{c}_1^{-\frac{{\alpha }_0}{2}}\right)+\frac{1}{\gamma {\Vert \nabla W\Vert }_{L^2}^2}{\rho }_1^{\frac{{\alpha }_2}{2}}\left(c_1^{\frac{{\alpha }_3}{2}}-c_1^{\frac{{\alpha }_2+{\alpha }_3}{2}}{c}_1^{-\frac{{\alpha }_2}{2}}\right)\\ =0.\end{array}$

Hence, we can obtain that

$f\left(c_2,\frac{c_2}{c_1}{\rho }_1\right)\ge f\left(c_1,{\rho }_1\right)\ge 0.$(14)

Moreover, we observe that if $g_{c_2}\left({\rho }^{\prime }\right)\ge 0$ and $g_{c_2}\left({\rho }^{″}\right)\ge 0$, then

$f\left(c_2,\rho \right)=g_{c_2}\left(\rho \right)\ge 0$ for any $\rho \in \left[{\rho }^{\prime },{\rho }^{″}\right]$.(15)

Really, if $g_{c_2}\left(\rho \right)<0$ for some $\rho \in \left({\rho }^{\prime },{\rho }^{″}\right)$ then there exists a local minimum point on $\left({\rho }_1,{\rho }_2\right)$ and this contradicts the fact that the function $g_{c_2}\left(\rho \right)$ has a unique critical point which has to coincide necessarily with its unique global maximum, see Lemma 3.2. By Equations (13) and (14), we can choose ${\rho }^{\prime }=\frac{c_2}{c_1}{\rho }_1$ and ${\rho }^{″}={\rho }_1$, and hence the Lemma follows.

By the Lemma 3.2 and Lemma 3.3, we can obtain that $f\left(c_0,{\rho }_0\right)=0$ and $f\left(c,{\rho }_0\right)>0$, for all $c\in \left(0,c_0\right)$ and ${\rho }_0:={\rho }_{c_0}>0$. So, we define

$B_{{\rho }_0}:=u\in H:{\Vert \nabla u\Vert }_{L^2}^2<{\rho }_0\text{and}V\left(c\right):=S\left(c\right)\cap B_{{\rho }_0}.$

The solution of the Hartree Equation (2) by variational method began with E.H.Lieb [29] , who proved that the solution of the equation is the solution is the following variational problem

$m\left(c\right):=\underset{u\in V\left(c\right)}{\text{inf}}E\left(u\right).$(16)

According to the above results, we have the following Lemma.

Lemma 3.4. For any $c\in \left(0,c_0\right)$, the following properties hold

(i)

$m\left(c\right)=\underset{u\in V\left(c\right)}{\text{inf}}E\left(u\right)<0<\underset{u\in \partial V\left(c\right)}{\text{inf}}E\left(u\right).$(17)

(ii) If $m\left(c\right)$ is reached, then any ground state is contained in $V\left(c\right)$

Proof. (i) For any $u\in \partial V\left(c\right)$, we have ${\Vert \nabla u\Vert }_{L^2}^2={\rho }_0$. Thus, according to Lemma 2.3, we have that

$E\left(u\right)\ge {\Vert \nabla u\Vert }_{L^2}^{2}f\left({\Vert u\Vert }_{L^2}^2,{\Vert \nabla u\Vert }_{L^2}^2\right)={\rho }_{0}f\left(c,{\rho }_0\right)>{\rho }_{0}f\left(c_0,{\rho }_0\right).$

Now, for any fixed $u\in S\left(c\right)$. For $\lambda \in \left(0,\infty \right)$, we set

$u_{\lambda }\left(x\right)={\lambda }^{\frac{N}{2}}u\left(\lambda x\right),$

Clearly $u_{\lambda }\in S\left(c\right)$ for any $\lambda \in \left(0,\infty \right)$. In the following calculation, let’s say $\lambda x=s,\lambda y=t$. We define on $\left(0,\infty \right)$ the map,

$\begin{array}{c}{\psi }_u\left(\lambda \right):=E\left(u_{\lambda }\right)\\ =\frac{1}{2}{\displaystyle {\int }_{{ℝ}^N}{\left|\nabla u_{\lambda }\right|}^2\text{d}x}-\frac{1}{p+2}{\displaystyle {\int }_{{ℝ}^N}{\left|u_{\lambda }\right|}^{p+2}\text{d}x}-\frac{1}{4}{\displaystyle {\int }_{{ℝ}^N}{\displaystyle {\int }_{{ℝ}^N}\frac{{\left|u_{\lambda }\left(x\right)\right|}^2{\left|u_{\lambda }\left(y\right)\right|}^2}{{\left|x-y\right|}^{\gamma }}\text{d}x\text{d}y}}\\ =\frac{1}{2}{\lambda }^N{\displaystyle {\int }_{{ℝ}^N}{\left|\nabla u\left(\lambda x\right)\right|}^2\text{d}x}-\frac{1}{p+2}{\lambda }^{\frac{N}{2}p+2}{\displaystyle {\int }_{{ℝ}^N}{\left|u\left(\lambda x\right)\right|}^{p+2}\text{d}x}\\ -\frac{1}{4}{\lambda }^{2N}{\displaystyle {\int }_{{ℝ}^N}{\displaystyle {\int }_{{ℝ}^N}\frac{{\left|u\left(\lambda x\right)\right|}^2{\left|u\left(\lambda y\right)\right|}^2}{{\left|x-y\right|}^{\gamma }}\text{d}x\text{d}y}}\\ =\frac{1}{2}{\lambda }^2{\displaystyle {\int }_{{ℝ}^N}{\left|\nabla u\left(s\right)\right|}^2\text{d}x}-\frac{1}{p+2}{\lambda }^{\frac{N}{2}p+2-N}{\displaystyle {\int }_{{ℝ}^N}{\left|u\left(s\right)\right|}^{p+2}\text{d}x}\\ -\frac{1}{4}{\lambda }^{\gamma }{\displaystyle {\int }_{{ℝ}^N}{\left|u\left(t\right)\right|}^2\left({\displaystyle {\int }_{{ℝ}^N}\frac{{\left|u\left(s\right)\right|}^2}{{\left|x-y\right|}^{\gamma }}\text{d}s}\right)\text{d}t}\\ =\frac{1}{2}{\lambda }^2{\Vert \nabla u\Vert }_{L^2}^2-\frac{1}{p+2}{\lambda }^{\frac{Np}{2}}{\Vert u\Vert }_{L^{p+2}}^{p+2}-\frac{1}{4}{\lambda }^{\gamma }{\displaystyle {\int }_{{ℝ}^N}{\displaystyle {\int }_{{ℝ}^N}\frac{{\left|u\left(x\right)\right|}^2{\left|u\left(y\right)\right|}^2}{{\left|x-y\right|}^{\gamma }}\text{d}x\text{d}y}}.\end{array}$

We consinder that

$\frac{Np}{2}<2\text{and}\gamma >2,$

we see that ${\psi }_u\left(\lambda \right)\to 0$ as $\lambda \to 0$. Therefore, there exists ${\lambda }_0>0$ small enough such that ${\Vert \nabla u_{{\lambda }_0}\Vert }_{L^2}^2={\lambda }_0^2{\Vert \nabla u\Vert }_{L^2}^2<{\rho }_0$ and $E\left(u_{{\lambda }_0}\right)=\psi \left({\lambda }_0\right)<0$. It shows that $m\left(c\right)<0$.

(ii) It is well known, see for example ( [30] , Lemma 2.7), that all critical point of E restricted to $S\left(c\right)$ belong to the Pohozaev’s type set

${\mathcal{Q}}_c:=\left\{u\in S\left(c\right):\mathcal{Q}\left(u\right)=0\right\}.$

Now, letting $u_{\lambda }\left(x\right)={\lambda }^{\frac{N}{2}}u\left(\lambda x\right)$

$\begin{array}{c}{\psi }_u\left(\lambda \right)=E\left(u_{\lambda }\right)\\ =\frac{1}{2}{\lambda }^2{\Vert \nabla u\Vert }_{L^2}^2-\frac{1}{p+2}{\lambda }^{\frac{Np}{2}}{\Vert u\Vert }_{L^{p+2}}^{p+2}-\frac{1}{4}{\lambda }^{\gamma }{\displaystyle {\int }_{{ℝ}^N}{\displaystyle {\int }_{{ℝ}^N}\frac{{\left|u\left(x\right)\right|}^2{\left|u\left(y\right)\right|}^2}{{\left|x-y\right|}^{\gamma }}\text{d}x\text{d}y}}.\end{array}$

Such that

$\frac{\text{d}}{\text{d}\lambda }E\left(u_{\lambda }\right)=\lambda {\Vert \nabla u\Vert }_{L^2}^2-\frac{Np}{2\left(p+2\right)}{\lambda }^{\frac{Np}{2}-1}{\Vert u\Vert }_{L^{p+2}}^{p+2}-\frac{\gamma }{4}{\lambda }^{\gamma -1}{\displaystyle {\int }_{{ℝ}^N}{\displaystyle {\int }_{{ℝ}^N}\frac{{\left|u\left(x\right)\right|}^2{\left|u\left(y\right)\right|}^2}{{\left|x-y\right|}^{\gamma }}\text{d}x\text{d}y}},$

and

${\frac{\text{d}}{\text{d}\lambda }E\left(u_{\lambda }\right)|}_{\lambda =1}={\Vert \nabla u\Vert }_{L^2}^2-\frac{Np}{2\left(p+2\right)}{\Vert u\Vert }_{L^{p+2}}^{p+2}-\frac{\gamma }{4}{\displaystyle {\int }_{{ℝ}^N}{\displaystyle {\int }_{{ℝ}^N}\frac{{\left|u\left(x\right)\right|}^2{\left|u\left(y\right)\right|}^2}{{\left|x-y\right|}^{\gamma }}\text{d}x\text{d}y}}.$

Therefore

$\mathcal{Q}\left(u\right)={\Vert \nabla u\Vert }_{L^2}^2-\frac{Np}{2\left(p+2\right)}{\Vert u\Vert }_{L^{p+2}}^{p+2}-\frac{\gamma }{4}{\displaystyle {\int }_{{ℝ}^N}{\displaystyle {\int }_{{ℝ}^N}\frac{{\left|u\left(x\right)\right|}^2{\left|u\left(y\right)\right|}^2}{{\left|x-y\right|}^{\gamma }}\text{d}x\text{d}y}}$

For any $v\in S\left(c\right)$ and any $s\in \left(0,\infty \right)$

${{\psi }^{\prime }}_v\left(\lambda \right)=\lambda {\Vert \nabla v\Vert }_{L^2}^2-\frac{Np}{2\left(p+2\right)}{\lambda }^{\frac{Np}{2}-1}{\Vert v\Vert }_{L^{p+2}}^{p+2}-\frac{\gamma }{4}{\lambda }^{\gamma -1}{\displaystyle {\int }_{{ℝ}^N}{\displaystyle {\int }_{{ℝ}^N}\frac{{\left|v\left(x\right)\right|}^2{\left|v\left(y\right)\right|}^2}{{\left(x-y\right)}^{\gamma }}\text{d}x\text{d}y}}$

$\begin{array}{c}\frac{1}{\lambda }\mathcal{Q}\left(v_{\lambda }\right)=\frac{1}{\lambda }\left[{\Vert \nabla v_{\lambda }\Vert }_{L^2}^2-\frac{Np}{2\left(p+2\right)}{\Vert v_{\lambda }\Vert }_{L^{p+2}}^{p+2}-\frac{\gamma }{4}{\displaystyle {\int }_{{ℝ}^N}{\displaystyle {\int }_{{ℝ}^N}\frac{{\left|v_{\lambda }\left(x\right)\right|}^2{\left|v_{\lambda }\left(y\right)\right|}^2}{{\left|x-y\right|}^{\gamma }}\text{d}x\text{d}y}}\right]\\ =\lambda {\Vert \nabla v\Vert }_{L^2}^2-\frac{Np}{2\left(p+2\right)}{\lambda }^{\frac{Np}{2}-1}{\Vert v\Vert }_{L^{p+2}}^{p+2}-\frac{\gamma }{4}{\lambda }^{\gamma -1}{\displaystyle {\int }_{{ℝ}^N}{\displaystyle {\int }_{{ℝ}^N}\frac{{\left|v\left(x\right)\right|}^2{\left|v\left(y\right)\right|}^2}{{\left|x-y\right|}^{\gamma }}\text{d}x\text{d}y}}.\end{array}$

By the above calculation, we can obtain that

${{\psi }^{\prime }}_v\left(\lambda \right)=\frac{1}{\lambda }\mathcal{Q}\left(v_{\lambda }\right),$(18)

Here ${{\psi }^{\prime }}_v$ denotes the derivative of ${\psi }_v$ with respect to $\lambda \in \left(0,\infty \right)$. Hence, for any $u\in S\left(c\right)$ can be written as $u=v_{\lambda }$ with $v\in S\left(c\right)$, satisfy ${\Vert \nabla v\Vert }_{L^2}=1$ and $s\in \left(0,\infty \right)$.

Since the set ${\mathcal{Q}}_c$ contains all the ground states, we infer from Equation (18) that if $\omega \in S\left(c\right)$ is a ground state, there exists a $v\in S\left(c\right)$, ${\Vert \nabla v\Vert }_{L^2}^2=1$, and a ${\lambda }_0\in \left(0,\infty \right)$ such that $\omega =v_{{\lambda }_0}$, $E\left(\omega \right)={\psi }_v\left({\lambda }_0\right)$, and ${{\psi }^{\prime }}_v\left({\lambda }_0\right)=0$. Namely, ${\lambda }_0\in \left(0,\infty \right)$ is a zero of the function ${{\psi }^{\prime }}_v$.

Now, since ${\psi }_v\left(s\right)\to 0^{-}$, ${\Vert \nabla v_{\lambda }\Vert }_{L^2}=1$, as $\lambda \to 0$ and ${\psi }_v\left(\lambda \right)=E\left(v_{\lambda }\right)\ge 0$ when $v_{\lambda }\in \partial V\left(c\right)=\left\{u\in S\left(c\right):{\Vert \nabla u\Vert }_{L^2}^2={\rho }_0\right\}$, ${{\psi }^{\prime }}_v$ must has a first zero ${\lambda }_1>0$ corresponding to a local minimum. In particular, $v_{{\lambda }_1}\in V\left(c\right)$, and $E\left(v_{{\lambda }_1}\right)={\psi }_v\left({\lambda }_1\right)<0$. Also, from ${\psi }_v\left({\lambda }_1\right)<0$, ${\psi }_v\left(\lambda \right)\ge 0$ when $v_{\lambda }\in \partial V\left(c\right)$ and ${\psi }_v\left(\lambda \right)\to -\infty$ as $\lambda \to 0$, ${{\psi }^{\prime }}_v$ has a second zero ${\lambda }_2>{\lambda }_1$ that satisfies the local maximum of ${\psi }_v$. Since $v_{{\lambda }_2}$ satisfies $E\left(v_{{\lambda }_2}\right)={\psi }_v\left({\lambda }_2\right)\ge 0$, we have that $m\left(c\right)\le E\left(v_{{\lambda }_1}\right)<E\left(v_{{\lambda }_2}\right)$. In particular, since $m\left(c\right)$ is reached, $v_{{\lambda }_2}$ is not a ground state.