A sensitive test for new physics is the processes of the top quark due to large mass. The predictions of the Standard Model (SM) for the top quark in flavor changing neutral (FCN) transitions are strongly suppressed [1] as a result of the Glashow-Iliopoulos-Maiani (GIM) mechanism [2] . However, rare decays with branching ratios (BR) of order 10⁻⁵ - 10⁻⁶ may be detectable, depending on the signal. Any hint for new top quark physics at LHC would motivate further study at the next generation of collider experiments [3] . Recent discovery of a SM-like Higgs boson with a mass near 125 - 126 GeV [4] [5] has generated new moti- vations to study the extended Higgs sector. The two-Higgs doublet model (2THDM) is one of the simplest extensions of the SM, adding a second Higgs doublet with the same quantum numbers as the first one [6] . The versions that involve natural flavor conservation and CP conservation in the potential through the introduction of a discrete symmetry, are known as 2HDM-I [7] [8] and 2HDM-II [9] . A general version which is named as 2HDM-III allows the presence of flavor-changing neutral scalar interactions (FCNSI) at a three-level [10] [11] . There are also some variants (known as top, lepton, neutrino), where one Higgs doublet couples predominantly to one type of fermion [12] , while in other models, it is even possible to identify a candidate for dark matter [13] [14] . The definition of all these models, depends on the Yukawa structure and symmetries of the Higgs sector, whose origin is still not known. The possible appearance of new sources of CP violation is another characteristic of these models [15] .

Within 2HDM-I where only one Higgs doublet generates all gauge and fermion masses, while the second doublet only knows about this through mixing, and thus the Higgs phenomenology will share some similarities with the SM, although the SM Higgs couplings will now be shared among the neutral scalar spectrum. The presence of a charged Higgs boson is clearly the signal beyond the SM. Within 2HDM-II, one also has natural flavor conservation [16] , and its phenomenology will be similar to the 2HDM-I, although in this case, the SM couplings are shared not only because of mixing, but also because of the Yukawa structure. The distinctive characteristic of 2HDM-III is the presence of FCNSI, which requires a certain mechanism in order to suppress them, for instance, one can impose a certain texture for the Yukawa couplings [17] , which will then predict a pattern of FCNSI Higgs couplings. Within all those models (2HDM-I, II, III) [18] , the Higgs doublets couple with all fermion families, with a strength proportional to the fermion masses, modulo other parameters.

In the present work, we calculate the the BR for the decay in the framework of the general 2HDM.

Given and two complex doublet scalar fields with hypercharge-one, the most general gauge invariant and renormalizable Higgs scalar potential is [19]

where, and, , , are real parameters and, , , and are complex parameters.

Now, the most general -conserving vacuum expectation values (vev) are

where we choose a basis in which and are real and non-negative, , and

In the literature, the sign of the vev is chosen positive for convenience, however it could also take negative sign. The mass of the gauge fields are proportional to the square of the vev. The fermions have proportional masses to the vev then to be defined positively would take Yukawa couplings negative. In this way we would obtain consistent models and equal prediction regardless of the sign of the vev.

In Equation (2), the phase of is eliminated by using a global hypercharge rotation. In Equation (3), the complex phase can be removed by redefining the complex parameters, , ,. Thus, the CP violation is explicit in the scalar potential. The neutral components of the scalar Higgs fields

in the interaction basis can be written as, where

denote the real part. The third neutral scalar field in the interaction basis defined as is orthogonal to the Goldstone boson for the Z boson. As a result of the explicit breaking for the CP symmetry a mixing matrix R for fields is generated. This matrix relates the mass eigenstates with fields as follows

where R can be written down as:

and, for and. The neutral

Higgs bosons are defined to satisfy the masses hierarchy given by the inequalities [20] .

For the THDM with no CP violation in scalar sector the and are mixed in a matrix and the mass eigenstates are CP-even while is not mixed and has CP-odd symmetry. In this case the, and are equivalent to neutral scalar H, h and psedoscalar A in the THDM type I and II, respectively. In the type I and II models with explicit CP violation in the parameters of scalar potential, the two Higgses, h and H, CP even are mixed with the pseudo-scalar, odd CP, generating mass eigenstates with undefined CP symmetry. In THDM type III, which corresponds to the more general model without discrete symmetries, the mixture between the odd and even CP scalar is natural and there are mixed via a rotation matrix with scalar fields that have no defined CP. For the Yukawa interactions between fermions and scalars fields the most general structure are

where are the Yukawa matrices. The and denote the left handed fermions doublets under meanwhile, , correspond to the right handed singlets. The zero superscript in fermions fields stands for non mass eigenstates. After getting a correct spontaneous symmetry breaking by using (2) and (3), the mass matrices become

where for. The matrices are used to diagonalize the fermions mass matrices and relate the physical and weak states

In order to study the rare top decay we are interested in up-type quarks fields. By using Equations (5), the interactions between neutral Higgs bosons and fermions can be written in the form of the 2HDM type II with additional contributions which arise from Yukawa couplings and contain flavor change. From now on, we will omit the subscript 1 in Yukawa couplings to simplify the notation. Therefore, the interactions for up-type quarks and neutral Higgs bosons are explicitly written as

where

and

The fermion spinors are denoted as. Note that Latin indices in (10) and (11) denote the three neutral Higgs bosons meanwhile Latin indices in spinors, Yukawa matrix or mass matrix are for flavor of the up-type quarks. The CP conserving case is obtained if only two neutral Higgs bosons are mixed with well-defined CP states, for instance for is the usual limit.

We are interested in the contributions of the flavor changing neutral scalar interactions to the rare top decay which come from previous Yukawa interactions. The dominant contributions for this decay at one loop coming from neutral Higgs; The charged Higgs contributions are suppressed by the bottom-quark mass compared to the top-quark mass in the neutral Higgs contributions [21] . For the partial width of the decay, using Equation (9), we have

where, at electroweak scale and the functions are defined as

with. In order to give the expression for branching ratio for the rare top decay we consider as an approximation to take the reported total width for top quark as [22] . Therefore, the branching ratio can be written as

The above expression contains too many free parameters of the model, such as the masses Last expression contains several free parameters of the THDM, such as the masses of neutral Higgs bosons and the mixing parameter and. In the next section the parameters are treated to study the rare top decay.

First we will discuss the free parameters involved in the process. The Yukawa couplings in the THDM-III are responsible for the FCNSI as shown the expression (9). One possible option to suppress these FCNSI is obtained by assuming an ansantz for the Yukawa couplings. We take into account the ansantz proposed by Cheng-Sher [23] . This ansatz assumes a specific structure for the Yukawa matrix given by.

THDM type III in Yukawa Lagrangian has two sectors: in one of them the couplings are proportional to the masses of the fermions and does not generate flavor changing. The other sector generates flavor changing at tree level. This situation occurs because the two Yukawa matrices can not be diagonalized simultaneously with one rotation. The mass of the fermions and the factor that generates flavor changing are a linear combination of the two Yukawa matrices of the Lagrangian. Depending on this linear combination to generate the fermion masses, THDM type I or THDM type II can be generated plus additional terms that produce the flavor changes. In general THDM type III produces four different types of Lagrangians making linear combinations of type I and II for the up and down quarks. Then, in THDM type III we can choose the sector without flavor changing as type II and add the respective flavor changing that appear in this model. For this reason, a sector of parameters THDM type II in the various processes analyzed in the literature as, , , etc.

For the masses of neutral scalar we set the mass of the lightest Higgs boson equal to the value of the mass of the observed scalar reported by ATLAS and CMS,. The masses of the and are fixed as 300 GeV and 600 GeV, respectively. If neutral scalar fields have greater values of masses, then their contribution to the will be negligible. Therefore, the set of the free parameters in the partial width (12) is reduced only to the mixing angles. In order to analyze the branching ratio for rare top decay we consider allowed regions for the mixing parameters and. The numerical results show that under above assumptions the branching ratio (15) does not have significant contributions from mixing parameter in the interval. Then, we just focus in the parameters. The considered regions for and are studied in previous work by the authors [24] . These allowed regions for the parameter space are obtained by experimental and theoretical constrains in the framework of the 2HDM type II with CP violation for fixed and the mass of the charged Higgs bosons [25] . One can obtain the following regions for and for, and:

and

Figures 1-3 show the behavior of the branching ratio for as function of and in the allowed regions, and, respectively.

The obtained for allowed, mixing parameters is of the order of. This branching ratio is obtained for and masses of the, neutral scalars greater than 600 GeV. We note that the first row in scalar mixing matrix, Equation (6), does not have the, which leaves the branching ratio independent of. The obtained limits for are less suppressed than the SM limits. These regions are restrictive for mixing parameter. In order to explore the behavior of the branching ratio for mixing parameter in an greater range we generate a set of random values for and and obtain Figure 4, which shows an accumulation of points in the values of for the branching ratio. We note that the contributions from FCNSI are greater than SM contributions [26] .

From 2015 to 2017, the experiment is expected to reach 100 fb⁻¹ of data with a energy of the center of mass of 14 TeV. In the year 2021, it is expected to reach a luminosity of the order of 300 fb⁻¹ of data. Experiments with this luminosity could find evidence of new physics beyond SM. Then, Run 3 in LHC could

observe events for the flavor changing neutral processes, which can be explained in a naive form as. For a luminosity of 300 fb⁻¹ and of the order of 176 pb with a on the order of 10⁻⁸, Figure 4, the number of events for is obtained as is seen in Figure 5 for the different values of the parameters, and according to the experimentally allowed regions.

Last experimental results have obtained a bound for these branching ratios such as [22] . If we fix the branching ratio (15) equal to the experimental upper bound, then lower bound for the parameter is constrained for any and the values of from, see Figure 6. We note that the branching ratio decreases as increase in the value of

. In the case of the scatter plot 4, we can estimate from 1 to 5 possible events for from to.

This work is supported in part by PAPIIT Project IN113916, Sistema Nacional de Investigadores (SNI) in México, and PIAPI 1628 Project. R. M. thanks to COLCIENCIAS for the financial support.

Gaitán-Lozano, R., de Oca, J.H.M. and Martinez, R. (2017) Rare Top Decay t ® cg in General THDM- III. Journal of Quantum Information Science, 7, 57-66. https://doi.org/10.4236/jqis.2017.72006