Exploring new resonances with direct top flavor changing interactions

Flavor-changing neutral current (FCNC) processes, which are highly suppressed in the Standard Model (SM) due to the well-known Glashow–Iliopoulos–Maiani (GIM) mechanism [1], are among the most powerful indirect probes of new physics (NP) beyond the SM. For this reason, over the past several decades, FCNC interactions involving “light” flavors have been stringently constrained by precision observables in flavor physics. On the other hand, the FCNC effects in the top-quark sector remain comparatively less explored and have attracted increasing attention in recent years [2−17]. There are two main methods widely used in the analysis of top-quark FCNC processes. One is to start from the complete set of effective operators in SMEFT and treating them without bias, commonly referred to as the model-independent approach. The other starts from specific NP models that contain heavy degrees of freedom that can induce FCNC processes in the top-quark sector.

The model-independent method, though very general and powerful for interpreting experimental results, retains little information about the underlying NP models behind the operators. To extract more information about the NP models from data, one needs to go a step further by considering operator correlations and relating specific new particles to characteristic patterns of operator correlations. In this work, we consider a comprehensive set of heavy scalar and vector bosons that are consistent with SM gauge symmetry and couple to the top quark via renormalizable up-sector flavor-changing interactions. Rather than treating SMEFT operators as arbitrary deformations, we derive them from explicit UV particle content, thereby enforcing correlations among operators that are otherwise invisible in purely model-independent analyses. This approach allows us to address questions that cannot be directly answered with generic SMEFT fits alone.

The paper is organized as follows. In Sec. II, we construct and classify the viable heavy bosons and define their interactions with SM quarks. In Sec. III, we perform the matching to SMEFT and discuss the associated LHC phenomenology. We conclude in Sec. IV with a discussion of the implications and outlook.

In this work, we consider new particles of spin 0 and spin 1 in NP models. For simplicity, we impose a set of requirements on them and on the new renormalizable interactions, as follows:

(1) The new particle must have well-defined quantum numbers under the SM gauge group;

(2) The new particle must couple to the top quark through direct top-flavor-changing interactions;

(3) There are no flavor-conserving interactions between the new particle and the SM quarks;

(4) The exotic particle couples to the SM quarks via only one type of interaction. ¹

Imposing these requirements, we list the possible new particles and interactions in Table 1, where $ Q_L $ denotes the left-handed quark doublet, $ u_R $ and $ d_R $ are the right-handed singlets, $ T^a $ and $ \tau^i $ are the generators of $ SU(3)_c $ and $ SU(2) $ in the fundamental representation, respectively, $ K_{\alpha \beta}^a $ is the Clebsch-Gordon (CG) coefficient for the symmetric coupling of two color triplets to a sextet, and $ \psi^c $ is the charge-conjugate of the fermion field ψ. One may have noticed that the $ Z_{R 1 \mu}^{{}'}(Z_{R 8 \mu}^{{}'}) $ and $ Z_{L 1 \mu}^{{}'}(Z_{L 8 \mu}^{{}'}) $ have the same quantum numbers, which means that the new fermion number conserved color singlet or octet vector bosons can couple to both the left-handed and the right-handed quarks. However, in this work we will consider one kind of interaction at a time for simplicity.

With these requirements, there remains some complexity in the flavor structure, which is important for inducing top-quark FCNC processes. We add some assumptions to limit the possible flavor structures of the interaction as follows:

● To avoid introducing additional contributions to CP-violation, which are beyond the scope of this work, we assume $ y_{ij} = y_{ji}\in{\bf{R}} $ for now.

● We will turn on two of the three independent parameters $ y_{12} $, $ y_{23} $, and $ y_{13} $ at a time. This is because, if all three parameters are nonzero, a simple triangle-loop diagram would generate flavor-diagonal interactions for the exotic particle.

● Moreover, since any new interactions with the bottom quark would be strongly constrained by flavor-physics observables, we do not consider new resonances that couple to left-handed quark fields. Therefore, we investigate in detail the collider phenomenology of $ Z_R^\prime $, $ G_R^\prime $, and $ \tilde S_R $.

To obtain the correct combination and relative strengths of the effective operators at the electroweak (EW) scale for a given heavy particle, we first integrate out the heavy particle at its mass (cutoff) scale λ using the covariant derivative expansion (CDE) [18] to obtain the dimension-six operators in the Warsaw basis of SMEFT [19], together with their Wilson coefficients, computed with the $ \mathrm{MatchingTools}$ package [20]. The relevant operator at the cutoff scale is

and the Wilson coefficients are

It has been shown in [21] that the resonances considered here are the only three that can generate a tree-level dim-6 $ u_{\mathbb{C}}^2u_{\mathbb{C}}^{\dagger2} $-type operator under our assumptions. We then evolve these operators down to the top-quark mass scale $ m_t $ using the renormalization group equations (RGEs).

where $ \gamma_{ij} $ is the anomalous-dimension matrix [22−24]. During the running, operator mixing can generate additional operators. Although these mixing-induced operators are suppressed by a loop factor compared to the tree-level four-fermion operators, they can still play a role and should be examined. In particular, we focus on the operators $ O_{\varphi u} $, $ O_{u\varphi} $, $ O_{\varphi q}^{(1)} $, and $ O_{\varphi q}^{(3)} $, which contribute directly to $ t \to u (c) + Z/h $ processes [25−27]. The explicit forms of these operators are

where either i or j denotes a top-quark field operator. The partial widths of top-quark decay processes induced by these operators are

After fixing the coupling coefficients $ y_{ij} $ and the cutoff scales, we employ the $ \mathrm{Wilson}$ package [28] to compute these running Wilson coefficients at the top-quark mass scale. For simplicity, we do not run the tree-level Wilson coefficients $ C_{uu,ijkl} $ in this work.

In this section, we investigate the phenomenology associated with the effective operators derived in Sec. II and analyze the behaviors of the three new resonances. For each new particle, we specify the theoretical inputs at the cutoff scale, at which only four-fermion operators are generated. At the electroweak (EW) scale, two kinds of operators are phenomenologically important: the four-fermion interaction $ O_{uu} $ and the purely RG-induced operators listed in Eq. (6). The four-fermion operators contribute significantly to single-top production, to the branching ratio of the top quark decaying to three-jet final states, and to the production of both same-sign and opposite-sign top quarks at colliders [29], while the RG-induced operators generate both single-top production signals and anomalous decay signals of the top quark, which can be searched for at a “top-factory” such as the LHC. On the other hand, the RG-induced operators also predict anomalous production channels at a Higgs factory [30]. We assess the main phenomenological constraints on these two kinds of operators at the LHC. All signal cross sections are generated with $ \mathrm{MadGraph5}\_{\mathrm {aMC@NLO}}$ [31] at leading order (LO) at parton level with the NNPDF2.3 LO parton distribution function (PDF) [32].

We separate the combinations of the FCNC coupling constants into three distinct patterns, $ y_{23} = 0 $, $ y_{31} = 0 $, and $ y_{12} = 0 $. Before delving into the phenomenological analysis, we first list the characteristics of the different patterns.

We begin by enumerating and analyzing the three distinct patterns.

The effective operators can be divided into three groups:

$ {\bf{Group-1:}} $ The operators in this group contain a single (anti)top-quark field operator. Therefore, they contribute to single top-quark production and rare top-quark decay ($ t\to jjj $) processes. They are as follows (we denote $ O_{\bar ij\bar kl} $ for $ O_{uu,ijkl} $ for simplicity now).

Since there is no interference between the SM process and the effective operators, the leading-order contribution to the single top-quark production cross section can be factorized as

where $ \sqrt{S} = 1 $ TeV for simplicity.

$ {\bf{Group-2:}} $ The operators in this group contain two top-quark field operators. Therefore, they contribute to top-quark pair production and same-sign top-quark pair production processes. They are:

Note that the color-sextet scalar will not lead to exotic same-sign top-quark pair production (which would correspond to a same-sign up-quark operator) at the LHC, because it is a complex field that can be assigned an “up-number” $ U(1) $ quantum number, thereby restoring the conservation of the up-quark number in the Lagrangian of the UV theory.

Since the leading-order contribution to the top-quark production cross section again comes from interference terms, one has

Since there is no same-sign top-quark pair production in the SM, the leading-order contribution is again $ {\cal{O}}(\Lambda^{-4}) $.

$ {\bf{Group-3:}} $ The operators in this group do not contain any top-quark field operators, but they may contribute to $ D^0-\bar D^0 $ mixing. They are

Again, the color-sextet scalar does not contribute due to up-number conservation.

The phenomenology of this case is nearly the same as in the first case, except for the difference between the parton distributions of the up and charm quarks. The interaction Hamiltonian and the corrections in the three groups are:

$ {\bf{Group-1:}} $ The operators are

The coefficients for single top quark production are

$ {\bf{Group-2:}} $ The operators are

Again the color sextet scalar will not lead to the exotic same-sign top-quark pair production due to the charm-number conservation.

The coefficients for top quark pair and same-sign top quark pair productions are

$ {\bf{Group-3:}} $ The operators in this group are exactly the same as the first case.

In this case, there is no contribution to single-top-quark production. The relevant interaction Hamiltonians for top-quark pair production are:

while the relevant interaction Hamiltonians for the same-sign top-quark pair-production process are

In this case, the color sextet does not contribute to same-sign top-quark pair production due to “top-number” conservation in the UV model. Since the operators proportional to $ y_{12}y_{13} $ do not interfere with the SM, at leading order we have

The contribution to the $ D^0-\bar D^0 $ mixing in this case is suppressed by an additional factor of $ (16\pi^2\Lambda^2)^{-1} $. The coefficients appearing in the cross sections for top-quark production processes at the 13 TeV LHC are listed in Table 2.

Before discussing collider phenomenology, we begin by examining the constraint from the $ D^0-\bar D^0 $ mixing. The $ |\Delta C = 2| $ Hamiltonian is

where $ C_{12} $ is the Wilson coefficient at the electroweak scale. Therefore, the new-physics contribution to $ \Delta m_D\equiv|m_{D^0}-m_{\bar D^0}| $ is

where the hadronic matrix element is [33]

with $ f_D = 211.5\pm2.3\pm1.1\pm0.8 $ MeV [34], $ m_D = 1864 $ MeV, and $ B_D = 0.82 $. The measured value of $ \Delta m_D $ is [35]

which yields a rough constraint on the Wilson coefficient and the cutoff scale:

Although in a more careful calculation the RG running of the effective operators should be included, it will not qualitatively change the conclusion, since the nonperturbative parameters used in this calculation contain large uncertainties and dominate the theoretical error. Therefore, for the vector bosons $ Z_R^\prime $ and $ G_R^\prime $, $ y_{12}/\Lambda^2 $ is highly constrained by $ D^0-\bar D^0 $ mixing. By contrast, the color sextet does not receive a strong constraint from this observable at leading order.

However, the color sextet with $ y_{12} = 0 $ can hardly be constrained by any other process. It is therefore worth using this observable to constrain the parameters. The relevant effect arises from the loop-induced dimension-6 effective operator $ O_{\bar uc\bar uc} $, which is $ {\cal{O}}(\Lambda^{-4}) $, and the limit is roughly

Here we have ignored the additional logarithmic factor $ \log(\Lambda^2/m_t^2) $ since its effect is $ {\cal{O}}(1) $ over a wide range of λ. This will not qualitatively affect the result, since the result is mostly affected by the theoretical uncertainty from nonperturbative QCD.

For the single top-quark production channel, the signal events contribute to the t-channel single-top production. The most precise result is given in [36]:

It should be kept in mind that we provide only a very rough estimate here for several reasons:

● Different combinations of initial-state partons affect not only the total cross section but also the rapidity distribution of the $ tq $ system. For example, the $ uu $ initial state produces more signal events at central rapidities.

● In the new-physics model, the produced top quark is right-handed; consequently, the charged lepton from its decay tends to be more energetic than that from a left-handed top quark. As a result, the cut acceptance for new-physics single-top-quark events is expected to be slightly higher than for SM t-channel single-top-quark events. However, the extent of this effect depends on the details of the kinematic selections and on the weight parameters of the artificial neural networks used in the analysis.

Nevertheless, a full detector-level simulation is beyond the scope of this work for this channel as well as for the opposite-sign and same-sign top-quark pair production channels. We leave these to future work. The SM predictions at NNLO in QCD are [37]

At 95% CL, we find

For the $ t\bar t $ production channel, the total cross section measured at the LHC is [38]

while the SM prediction is [39]

At 95% CL, we obtain

Comparing with the numbers in Table 2, one finds that the single-top channel is more sensitive to the new physics considered here than the $ t\bar t $ channel.

Same-sign top-quark pair production is another important channel for the new resonances discussed in this work. For this channel, the observed upper limit on $ \sigma(tt) $ is 1.6 fb at 95% CL [40] ². It is obvious that the strongest constraints on $ Z^\prime_R $ and $ G^\prime_R $ come from the $ tt $ channel.

The limits on the branching ratios of rare top-quark decays are [15, 41, 42]

Our numerical results show that these observables yield weaker constraints in all cases. Combining the constraints from the LHC and $ D^0-\bar D^0 $ mixing, the allowed parameter regions are shown in Figs. 1-3.

Figure 1.  (color online) The allowed regions in the parameter space for $Z^\prime_R$ are shown. The light-red region is constrained by the same-sign top-quark production cross section at the 13 TeV LHC. The gray region is constrained by the $D^0-\bar D^0$ mixing observable. The constraint from $D^0-\bar D^0$ mixing is omitted when it is always weaker than that from top physics.

Figure 2.  (color online) Shown are the allowed regions in the parameter space for $G^\prime_R$. The light red region is constrained by the same-sign top-quark production cross section at the 13 TeV LHC. The gray region is constrained by the $D^0-\bar D^0$ mixing observable. The constraint from $D^0-\bar D^0$ mixing is omitted if it is weaker than that from top physics.

Figure 3.  (color online) Allowed regions in the parameter space of $\tilde S_R$. The light red region is determined by the single-top-quark production cross section at the 13 TeV LHC. The gray region is determined by the $D^0-\bar D^0$ mixing observable. The constraint from $D^0-\bar D^0$ mixing is omitted when it is everywhere weaker than that from top physics.

In this work, we explore the landscape of heavy resonances $ Z^\prime_R $, $ G^\prime_R $, and $ \tilde S_R $ that couple the right-handed top quark directly to the right-handed light up quark in the UV model. We derive the effective operators induced by these new particles at the EW scale and present a preliminary phenomenological analysis. Figs. 1-3 clearly show that the phenomenology of $ \tilde S_R $ differs markedly from that of $ Z^\prime_R $ and $ G^\prime_R $. This is not surprising, since the fermion-number-violating resonance $ \tilde S_R $ is more exotic than the fermion-number-conserving $ Z^\prime_R $ and $ G^\prime_R $, and therefore does not predict a pronounced $ tt $ signal at the LHC. Accordingly, an excess of exotic single-top events without a same-sign-top excess would favor the sextet scalar, while an excess of exotic same-sign-top events without a single-top excess would favor the singlet and octet vector bosons.

On the other hand, it is difficult to distinguish between $ Z^\prime_R $ and $ G^\prime_R $. Although the effective operators and Wilson coefficients differ in these two cases, the interactions in the UV model share the same Lorentz structure; the differences arise from the color structure and associated color factors. To extract this information, measuring the ratio $ \delta\sigma(tt)/\delta\sigma(tj) $ is necessary. Nevertheless, further work is required to reduce the theoretical uncertainties.

Our analysis also shows that phenomenology can effectively discriminate among patterns of flavor-changing couplings in the UV model. For $ Z^\prime_R $ and $ G^\prime_R $, whenever $ y_{12}\neq 0 $, there is a significant contribution to $ D^0-\bar D^0 $ mixing, leading to a strong constraint. This makes $ D^0-\bar D^0 $ mixing, together with the same-sign-top signal at the LHC, a powerful tool to probe the coupling patterns of $ Z^\prime_R $ and $ G^\prime_R $. Conversely, if an excess is observed in both single-top and same-sign-top events in the near future, it would favor the $ y_{12} = 0 $ pattern. For the color sextet in the $ y_{12} = 0 $ pattern, although the constraint from $ D^0-\bar D^0 $ mixing is much weaker due to loop suppression, it still provides valuable information, since other constraints are even weaker.

To distinguish the $ y_{13} = 0 $ pattern from the $ y_{23} = 0 $ pattern, it suffices to note that the initial-state partons differ substantially in these two cases. This difference affects not only the predicted total cross section, but also the event shape: the presence of a valence u quark tends to produce more energetic same-sign-top events via the $ uu $ initial state, and a broader rapidity distribution for the single-top process via the $ u\bar u $, $ u\bar c $, and $ uc $ initial states.

We would like to thank Dr. K. Chai and Dr. R. Zhang from the Theoretical Physics Division of the Institute of High Energy Physics (IHEP), Chinese Academy of Sciences, for their valuable assistance.