Developments in wireless communication systems and increasing demand for data transmission, cause push communication systems to acquire more and more spectral resources. Thus, fixed spectrum allocation is not an efficient technique for new communication networks. Cognitive radio is a novel solution to overcome this limitation. Users in cognitive radio networks are secondary users (SU) who utilize spectrum opportunities for communication in the absence of a primary user (PU) who is the licensed owner of a frequency band. In this thesis, we investigate the spectrum sensing challenges in cognitive radio networks. Knowledge about PUs presence/absence is a key problem because missing a PU results in interference and jeopardizes its performance. Beginning with single user spectrum sensing methods, we show that the cooperative spectrum sensing outperforms the former approach considerably. Then we study challenges of cooperative spectrum sensing in cognitive radio networks. Cooperative spectrum sensing methods utilize more available resource e.g. bandwidth and power. We describe local sensing methods like energy, cyclostationary and eigenvalue based detection that are used in cooperative spectrum sensing in cognitive radio networks and evaluate their advantages and short comings. Among these schemes, energy detection and eigenvalue based methods, need no information about PU signal. Energy detection is the simplest one but it needs the noise variance and shows poor performance if the estimated noise power is inaccurate, Therefore it is not efficient for practical systems with a low PU SNR regime e.g. received SNR is about -20dB from a wireless microphone operating in TV bands several hundred meters away. Random matrix theory is a modern solution for complicated stochastic problems which studies stochastical behavior of random matrix. Random matrix theory provides asymptotical solutions to evaluate detection threshold. In this work we introduce popular random matrix like Gaussian matrix and Wishart matrix and study statistical distribution of random matrix eigenvalues and their extremes. We analyze the eigenvalue based spectrum sensing methods after the random matrix theory descriptions. In a cognitive radio network, detection perforamance could be enhanced by increasing number of cognitive users, however, more available resources like bandwidth and power are needed. On the other hand, increasing the number of SUs leads to better performance of cooperative spectrum sensing, however, total probability of error has a lower bound and increasing the number of SUs cannot decrease the bound further. Due to this tradeoff between detection performance and available resource usage, we investigate optimum number of cognitive users for cooperation and show that the maximal cooperation in spectrum sensing is not optimal. However, it is very important to select the appropriate set of these users for cooperative spectrum sensing. Due to this issue, user selection algorithms are investigated and a new one is proposed. We show that it outperforms random selection algorithm and decreases the computational complexity of eigenvalue based methods. We find the optimum number of users for cooperative spectrum sensing via Monte-Carlo simulations in MATLAB for several scenarios and show the improvement. Keywords ; Cooperative spectrum sensing, Random matrix theory, Eigenvalue based spectrum sensing, User selection, Cognitive radio