according to the importance of planetary gearboxes and their increasing use, it is necessary to pay attention to the knowledge needed to monitor their condition and health in industry. In the present study, using the potential energy method, the mesh stiffness of gears, which is considered as the main source of vibration inside the gearbox, was extracted for both healthy gear and cracked gear. Next, by using a lumped mass dynamic model with 21 degrees of freedom for the planetary gearbox, the healthy and faulty gearbox system was simulated. The results of transverse and torsional vibration signals of the system were examined. Transverse vibration signals have a complex spectral structure caused by time variant gears meshing locations due to the rotation of planet carrier and sun gear. They include amplitude modulation terms due to the vibration transfer path that give no extra information about gear faults, but increase the spectral complexity and make it difficult to diagnose faults. Torsional vibration signals are free from the extra modulation effect and have simpler spectral structure and they are better to reveal gear faults. Therefore, using the frequency analysis and demodulation analysis (Hilbert-Huang transform) of the extracted signals, the defected gear was detected. A laboratory system designed and built to perform validation tests for future works. Keywords planetary gearbox, dynamic model, torsional vibration, demodulation analysis, fault diagnosis