Direction of arrival (DOA) estimation of the signal sources is one of the most challenging and practical issues in many areas such as wireless communications, radar, radio astronomy, sonar and navigation. In this thesis we investigate the direction of arrival estimation algorithms in the array antennas. Among these methods, the suace-based algorithms are most well known for good performance with respect to the computational complexity. This thesis will focus on these algorithms, especially the multiple signal classification (MUSIC), ESPRIT and Root-MUSIC method. These algorithms mainly use the maximum likelihood estimation of the covariance matrix, so-called sample covariance matrix (SCM). It can be argued that the accuracy of direction of arrival estimation depends on the accuracy of the SCM estimation. It is important to note that the sample covariance matrix estimation is not accurate and reliable in many practical situations, e.g. when the number of snapshots (N) and the number of antenna array elements (M) are both large and of the same order. Under these conditions, the random matrix theory claims to make a more reasonable approximation to the distribution function of the eigenvalues of the SCM by analyzing the behavior of this matrix in this asymptotic regime. This thesis aims to improve the suace based algorithms under the new asymptotic regime and the noise types with heavy-tailed distributions such as impulsive noise conditions. However, the performance of the SCM and resulting the direction-of-arrival estimation degrade in nonGaussian noise. In this thesis at the first we use the Marrona Estimator to improve the ESPRIT and Root-MUSIC algorithms and the next we use convex optimization methods to improve the DOA estimation algorithm, G-MUSIC, by modifying the eigenvector and eigenvalue estimation of the sample covariance matrix under non-Gaussian noise conditions. Simulation results confirm this performance improvement. Key Words: Direction of Arrival (DOA), MUSIC , G-MUSIC, Impulsive Noise