In the recent years, active magnetic bearings ( AMBs ) are often used in high speed rotating machinery for their advantages in comparison with conventional bearings. Some advantages of implementing the AMB system include the contactless operation, complete elimination of oil-based lubrication system, high control precision, wide operational temperature ranges, lower maintenance costs and longer system life. However, the AMB system is highly nonlinear in nature that causes nonlinear phenomena in the rotor motion. So, the major purpose of this research is to predict the nonlinear dynamic behavior of a coaxial rotor system supported by two AMBs and contact with two auxiliary bearings. In many cases of rotating machines, like jet engines, coaxial shafts are used for power transmission between a high/low pressure turbine and a compressor. Because of bearings nonlinearity and interaction between the inner and outer shaft, nonlinear dynamic analysis of these machines is rather complex. In this study, the coaxial rotor- AMB is modeled by ten degrees of freedom in x-z and y-z planes. In this model, the gyroscopic moments of the disks, flexibility of the inner shaft and geometric coupling of the magnetic poles are included. The nonlinear differential equations of motion are developed by the Lagrange’s equations and solved using the fourth order Rung-Kutta method in MATLAB software. Nonlinear dynamic analysis of system is investigated by means of the bifurcation diagrams, dynamic trajectories, power spectra, Poincare´ maps and maximum Lyapunov exponent. The effects of the rotational speed of the shafts, gravity, inner shaft stiffness, inter-rotor bearing stiffness, auxiliary bearings stiffness, disk position on the inner shaft, unbalance, controller gains of AMBs , clearance and friction in auxiliary bearings on the dynamic behaviour of the coaxial rotor- AMB system are studied. The results indicate that dynamics of the system can be significantly affected by varying these parameters, so that the system responses reveal a rich variety of nonlinear dynamical phenomena including period- 2 , - 4 , quasi-periodic and chaotic vibrations, as well as jump phenomena. Also, some threshold values are obtained with regard to the design of appropriate parameters for the system. Therefore, these results can play an important role in improving the performance of the coaxial rotor- AMB systems with auxiliary bearings. For the experimental verification of the results and careful attention to design aspects, a rotor – AMB test device with auxiliary bearings is designed and constructed. Then, the model for the test device is presented. The simultaneous incorporation of the gyroscopic moments of disk, flexibility of shaft, geometric coupling of magnetic actuators and auxiliary bearings contact forces is novelty of this model. The effects of rotational speed, disk unbalance and auxiliary bearings stiffness on the dynamic behaviour of the system are investigated. The results show that the increasing unbalance delays the occurrence of chaotic behavior, although smaller periodic motion range is accessible as a result. The smaller auxiliary bearings stiffness can lead to larger periodic motion range. With decreasing auxiliary bearings stiffness, the first region of the chaotic motion occurred at lower rotational speed. For the first through third experimental tests, a mass unbalance is added at three radial positions to the center of the disk, respectively. The fourth through sixth experimental tests are carried out for three deferent values of the auxiliary bearings stiffness. The other physical parameters are held constant in each experiment. Finally, the rotor orbits and power spectra obtained using the numerical simulations and the experimental tests are compared. Keywords: Coaxial rotor, AMB system, auxiliary bearing, nonlinear dynamic, chaos, experimental verification.