LAUSR

We present a theoretical study of transverse mode instability (TMI) in non-circular ytterbium-doped fibers including the rectangular core in a circular or D-shaped cladding. The D-shaped cladding is found efficient to suppress the TMI thanks to better heat dissipation, as compared to the circular cladding. However, the rectangular core does not suppress the TMI despite its better heat dissipation than a circular core counterpart. Although the temperature built in the rectangular core decreases with an increasing aspect ratio of the rectangular core, the low temperature does not benefit the TMI suppression. Instead, the TMI becomes stronger than its circular core counterpart. Our study reveals that the power coupling between two involved modes and gain saturation effect play a significant role in influencing the TMI. The power coupling strength is associated with the frequency offset between two modes, and it grows with an increasing aspect ratio of rectangular cores, suggesting the longer axis of rectangular core promotes the TMI.