The fiber is a circularly symmetrical dielectric optical waveguide. There are four vector modes, including TE mode, TM mode, HE mode and EH mode [15]. Among them, the HE and EH mode both have odd and even mode states, and there is a phase difference of ±π/2 between the two mode states [16]. Generally, the effective refractive index difference between vector modes is small, and the modes will occur crosstalk during propagation to form a LP mode [17] [18]. Similar to the LP mode, the OAM modes are also composed of vector modes in the fiber. When the effective refractive index difference between the modes is greater than 1 ´ 10⁻⁴, the vector modes will be superimposed to generate the OAM mode.

The OAM modes are generally composed of the odd-mode and even-mode linear superposition of the same order HE mode or EH mode in the fiber, which can be expressed as [19]:

OAM ± l , m ± = HE l + 1 , m e v e n ± j HE l + 1 , m o d d (1)

OAM ± l , m ∓ = EH l − 1 , m e v e n ± j EH l − 1 , m o d d (2)

where l represents the topological charge, and m means the radial mode order. The superscript “±” of OAM ± l , m ± means the spin state of the OAM mode and the subscript “±” means the direction of the wavefront rotation. The “even” and “odd” represent even mode and odd mode, respectively.

According to Equation (1) and Equation (2), it is known that HE_l+1,m or EH_l-1,m can be synthesized into the OAM_l,m mode. The OAM_l,m mode formed by the superposition of HE l + 1 , m e v e n and HE l + 1 , m o d d has the same circular polarization and field rotation direction, while the OAM_l,m mode formed by the superposition of EH l − 1 , m e v e n and EH l − 1 , m o d d has the opposite behavior.

For effectively separate the vector modes and generate multiple OAM modes, the basic objectives of the fiber structure design [12] mainly include: 1) as many the OAM modes as possible can be generated, and the effective refractive index differences between the vector modes should large enough (>1 ´ 10⁻⁴). 2) The fiber structure should be matched with the annular distribution of the OAM mode field, and it has a excellent fiber performance in C + L band.

In this study, the PCF structure is improved from two aspects of fiber cladding air holes and fiber structure. The cladding air holes are set in a circular annular distribution around the fiber core, which is beneficial to the generation of OAM modes of annular mode field distribution. In addition, the cladding air hole is set to a combination of rectangle and circle in the structure to provide a higher air filling rate. This method not only ensures a large refractive index difference in the fiber, but also prevents the leakage of higher-order modes. when the cladding of the PCF uses a rectangular air hole, the structure is not easily deformed, and the PCF has more stable characteristics [20] [21]. With the development of technology, rectangular air holes can be realized by methods such as sol, gel, casting and photolithography [22] [23], and the PCF with rectangular air holes can be fabricated.

Figure 1 shows the distance from the core to the cladding air hole is unchanged, and the effect of the core radius on the topological charge, radial order and the mode effective refractive index difference between the OAM modes at a wavelength of 1.55 μm. As shown in Figure 1(a), when the core radius r₀ gradually increases, the second-order radial mode disappears gradually. When the core radius r₀ is 8 μm, all the second-order radial modes disappear completely. In order to facilitate modular division multiplexing, this study only considers the first-order radial mode, taking m = 1. When the core radius r₀ changes in the

range of 6.0 to 9.5 μm, the number of OAM modes increases at first and then decreases gradually, and the field distribution of the vector mode becomes more and more clear, which is beneficial to distinguish the vector modes of each order. When the core radius r₀ is 8.9 μm, the number of OAM modes is the highest and reaches the peak value. Figure 1(b) shows the relationship between the core radius and the effective refractive index difference of the vector modes. It can be seen that the difference between the vector modes becomes larger with the increase of the core radius r₀. Combining Figure 1(a) and Figure 1(b), considering both the number of OAM modes and the effective refractive index difference between vector modes, the core radius r₀ is 8.9 μm. Therefore, in the optimized design of the fiber structure, the effective refractive index differences between the vector modes cannot be blindly increased, and the number of OAM modes must be considered.

The cross-section of the improved PCF structure is shown in Figure 2. The fiber core is a large air hole with the air hole radius is r₀ = 8.9 μm and the thickness of annular high refractive index region is d = 2.5 μm. The cladding is arranged as four layers of air hole ring layer outward around the center of the circle, and the distance between cladding layers is Λ = 0.3 μm. The length of rectangular air hole is a = 1.5 μm, the width is b = 1.2 μm, while the diameter of the circular air hole is d₁ = 1.8 μm. The substrate material is silica (SiO₂), whose refractive index is n₁ = 1.444 at 1.55 μm, and air hole is n₂ = 1.

Figure 3 and Figure 4 show the normalized intensity distributions and the field intensity distribution in the Ez direction of the partial vector modes in the PCF at the wavelength of 1.55 μm, respectively. Figure 3 shows that the modes in the

fiber are confined to the annular region among the air holes, and both are vortex beams with zero light intensity at the center, but it is difficult to distinguish whether it is EH mode or HE mode. Combined with Figure 4, the EH mode and HE mode of each order can be clearly distinguished. The red and blue area pairs in Figure 4 represent the size of the mode field strength. It can be seen from Figure 4 that there are 12, 6 and 3 groups of red and blue area pairs of HE_12,1, HE_6,1 and HE_3,1 mode. EH_12,1, EH_6,1 and EH_3,1 mode also have this kind of field distribution. In the field intensity distribution diagram of the Ez direction, the EH modes are closer to the inner core boundary of the fiber, while the HE modes are closer to the cladding of the fiber [14].

According to Equation (1) and Equation (2), the mode field and phase distribution of the OAM mode generated by the odd mode and even mode superposition of HE mode are as shown in Figure 5. The first and second rows in Figure 5 are mode field and phase distributions of OAM 11 + mode ( OAM 11 + = HE 21 even + j HE 21 odd ) and OAM 31 + mode ( OAM 31 + = HE 41 even + j HE 41 odd ), respectively. We can see from Figure 5 that the annular distribution of intensity of the OAM mode remains unchanged. The area of the singularity in the center of

the aperture increases as the topological charge increases, and the aperture becomes thinner. Meanwhile, with the increase of topological charge l, the phase changes to 2πl with a period.

Based on the properties of topological charge and rotation polarization of the OAM mode, the vector modes existing in the PCF are analyzed. It can be known that the topological charge of the vector mode supported by the PCF is l = 1 - 13. When the topological charge is l = 1, the OAM_l,m mode has the same circular polarization and field rotation direction, and 2 OAM modes (OAM_±1,1 mode is formed by HE_2,1 mode) can be synthesized. When the topological charge number l ≥ 2, the OAM_l,m mode can be divided into four kinds of information states in which the topological charges are positive and negative and the polarization direction is left and right rotation. Therefore, the PCF structure can contain 50 OAM modes at the wavelength of 1.15 to 2.0 μm (850 nm). Compared with the 26 modes in reference [12] and the 46 modes in reference [14], the bands are wider and the number of modes is larger. Table 1 shows OAM modes supported by the PCF and the corresponding vector modes.

Figure 6 shows the effective refractive index of the vector mode varies with wavelength. We can know that as the wavelength increases, the mode effective refractive index n_eff decreases in the PCF. In addition, with the increase of the order of the OAM modes, the effective refractive index of the mode decreases

faster and the curve is more inclined. Figure 7 shows the effective refractive index differences Δn_eff between the vector modes varies with wavelength. When the wavelength increases, the effective refractive index differences Δn_eff become increasingly lager. As can be seen from the standard line with a refractive index

difference of 1 × 10⁻⁴ in Figure 7, the differences Δn_eff between the vector modes are greater than 1 × 10⁻⁴. In particular, the effective refractive index differences Δn_eff of HE_3,1 and EH_1,1 modes can reach 2.6 × 10⁻³ at the wavelength of 1.55 μm. The larger effective refractive index differences can effectively reduce the modal coupling and form a stable OAM mode transmission without serious crosstalk and mutual coupling.

The fiber dispersion is one of the important parameters in optical waveguides, which has a great impact on the performance of fiber, and it will cause the broadening of optical pulses and further increase the communication rate. PCF can adjust the various structural parameters to make the dispersion value of the photonic crystal fiber maintain a flat characteristic in a relatively wide band. The waveguide dispersion can be expressed as [19]:

D = − λ c ∂ 2 | Re ( n e f f ) | ∂ λ 2 (3)

where Re(n_eff) is the real part of mode effective refractive index.

Figure 8 shows the dispersion versus wavelength curve for each vector mode of the PCF at the wavelength of 1.15 to 2.0 μm (850 nm). It can be seen from Figure 8 that when the incident light wavelength is in the range of 1.15 to 2.0 μm (850 nm), the dispersion of the low-order mode (l < 6) have little change with increasing of wavelength, and the dispersion distribution curve tends to be flat. This phenomenon is beneficial to the stable transmission of the OAM modes in the PCF. The dispersion of the higher-order mode is larger than that of the lower-order mode, and becomes larger as the wavelength increases. This also verifies the unstable transmission of the high-order mode in the fiber. In addition, the dispersion of HE_3,1 mode is 46.9649 ps∙(nm∙km)⁻¹, and the dispersion of HE_4,1 mode is 57.4461 ps∙(nm∙km)⁻¹ at the wavelength of 1.55 μm.

When the light transmits through the core, it will leak into the air holes of the

cladding in the fiber, resulting in confinement losses in the PCF. The confinement loss L can be expressed as [24]:

L = 2 π λ 20 ln 10 Im ( n e f f ) (4)

where Im(n_eff) is the imaginary part of mode effective refractive index, and the unit of confinement loss L is dB∙m⁻¹.

The confinement loss L of the modes at the wavelength of 1.55 μm is shown in Table 2. We can know that each vector modes of the PCF has a relatively low confinement loss at a wavelength of 1.55 μm, and confinement loss is in the range of 10⁻¹¹ dB∙m⁻¹ to 10⁻⁹ dB∙m⁻¹. This is mainly because each order mode can be well constrained in the fiber core, so the overall confinement loss is relatively ideal. In addition, the confinement loss of HE_4,1 mode is only 1.30 ´ 10⁻¹⁰ dB∙m⁻¹, whose result is four orders of magnitude lower than that in reference [25]. Therefore, it can be seen that the structure of the PCF designed in this study has a small confinement loss when transmitting the vortex beam, which lays a foundation for the stable transmission of the vortex beam in the fiber in the experiment.

The nonlinear coefficient and the effective mode area in the fiber are also important factors influencing the mode transmission. The smaller the nonlinear coefficient is, the better the transmission performance of the fiber communication system. The nonlinear coefficient and the effective mode area are respectively expressed as [26].

γ = 2 π n 2 λ A e f f (5)

A e f f = ( ∬ | E ( x , y ) | 2 d x d y ) 2 ∬ | E ( x , y ) | 4 d x d y (6)

where E(x, y) is the distribution of electric field. Since the cladding material is silicon, n₂ = 2.3 ´ 10⁻²⁰ m²∙w⁻¹ is the nonlinear refractive of the fused silica.

The effective mode area and nonlinear coefficient of vector mode are shown in Figure 9 and Figure 10, respectively. It can be known from Figure 9 that as the wavelength and mode order increase, the mode effective area increases because the field strength of the higher-order mode is more likely to leak into the air cladding. On the contrary, the nonlinear coefficient in Figure 10 has the opposite change. The nonlinear coefficients of all vector modes in the PCF are small, ranging from 0.6 W⁻¹∙km⁻¹ to 1.5 W⁻¹∙km⁻¹. This phenomenon is beneficial to the OAM mode transmission in the fiber. The nonlinear coefficient of HE_8,1 mode at wavelength 1.55 μm is 0.80175 W⁻¹∙km⁻¹. Compared with in references [12] and [27], the nonlinear coefficient is smaller and the nonlinear effect is weaker in this study. Therefore, the better the transmission performance of the fiber communication system in this PCF will be.