Nowadays experimental modal analysis has found a wide range of applications in dynamic improvement of structures, enhancing analytical models, optimal dynamic design, vibration control, fault detection and structural health monitoring in numerous fields of mechanical, aerospace and civil engineering. The modal test is normally performed at laboratory where the boundary conditions are replicated and the system is excited by artificial forces. By processing the data acquired from system’s response and input force measurements, frequency response functions are computed through which different modal analysis methods enable to obtain modal parameters. Experimental vibration analyses which concerns large industrial or masonry structures, encounter technical difficulties and limitations. For instance, transferring a bridge for this purpose to the laboratory site is neither practical nor plausible in financial terms. Furthermore, measuring the system’s frequency response in its location will necessitate shutting it off and also eliminating mechanical and electrical disturbance noises, not to mention the difficulty of applying excitation forces in such conditions. On the other side, the system operational conditions are not identical to those present in the test and hence the results are not appropriate to our purpose. In order to alleviate these limitations, the operational modal analysis (OMA) has been developed. This method is purposed to obtain modal parameters by measuring vibrations under the influence of ambient forces in normal work conditions of the system. One of the obvious advantages of this method comparing to the conventional modal analysis resides in the dispensability of using specific excitation sources, recreating actual boundary conditions or stopping the system’s normal operation, hence enhancing the speed and reducing the cost. In this method, the model linearized about the actual force and boundaries of the whole system (and not a separated part of it) is obtained. Modal analysis of rotary machines and bearing structures finds many applications in investigating induced vibrations complications but is not applicable via performing conventional experimental modal tests because of above-mentioned limitations. OMA can be suitable for the purpose of obtaining the modal parameters of rotary machines. The vibration response of rotary machines contains a multitude of powerful harmonics which cause problem to the method as the excitation force is essentially assumed as white noise. In order to rectify the OMA method to deal with these harmonics present in the excitation source, these latter are identified in a first attempt and their influence on the resulting modal parameters are investigated. Another method may consist of splitting the response signals into stochastic and periodic parts then using the former for further analysis. In this thesis, different indexes for identifying harmonic components are investigated in order to reach a more sensitive and accurate method. Methods based on frequency domain decomposition (FDD), consisting of EFDD and CFDD, are studied and implemented for obtaining the modal parameters. In these methods, the harmonics parts identified via the harmonic index are eliminated from the frequency response by performing linear interpolation. For enhancing the performance, a novel MCFDD method has been proposed. Another method presented in this project for getting rid of the harmonic excitation forces consists of separating the response signals before performing the operational modal analysis. The stochastic part of the signal is used for obtaining the modal parameters via the FDD method in the frequency domain and SSI method in the time domain. Accuracy and effectiveness of existing and proposed methods are investigated and compared through computer simulations. Then their effectiveness has been evaluated in practical tests performed on a steel beam and a large industrial fan in Mobarakeh Steel Complex. Both simulations and tests show enhancements in the results in comparison to other methods. Keywords: Operational modal analysis, harmonic excition, rotating machinery, signal separation.