With the rapid expansion of advanced communications and multimedia services, the fifth generation (5G) of wireless communications requires new technologies that can simultaneously provide services for larger number of users with far greater throughput and more reliable transmission. One of the important technologies is massive MIMO system, in which a base station with a large number of antennas provides services for many users within the same time and frequency resources. It has been shown that this system can provide high spectral efficiency as well as high energy efficiency. On the other hand, dividing the coverage area of a macrocell into some small cells reduces the distance between the users and the base stations, resulting in a significant reduction in the power consumption of the network. Recently, in order to meet the requirements of the 5G wireless communications, the combination of massive MIMO systems with small cell systems and millimeter wave domain under the name of two-tier massive MIMO heterogeneous systems has been proposed as a promising solution. In these systems, a number of users are covered by the macro base station and other users are covered by the small cell base stations in each cluster, and also, in each cluster, one of the base stations is selected as the base station head for connecting to the macro base station. Unfortunately, in these systems, the simultaneous transmission of signals from the macro base station and the small cell base stations creates inter-tier interference in addition to intra-cell interference. To resolve these types of interference, in a recent study, a precoding method called Block Diagonalization (BD) that can be applied separately at the macro base station and small cell base stations has been presented along with some optimal power allocation algorithms. In this thesis, at first, the performance of BD precoding method is compared with ZF precoding method when applied at small cell base stations at each cluster considering the power allocation algorithms. Then it is shown that the BD method provides better spectral efficiency for macro users which are in the interference area of a cluster compared to ZF method. In order to deal with the high complexity of the hardware of the macro base station due to having large number of antennas, in this thesis, it is suggested to select a best subset of the antennas based on a suitable criteria, while maintaining the system's performance. Also, in the small cell base stations, the performance can be improved by selecting the optimal subset of antennas. The spectral efficiency of the system is investigated in the presence of antenna selection considering different power conditions of the macro and small cell base stations as well as the different number of antennas at macro base stations. Also, the selection method of the base station head among the available small cell base stations at each cluster is considered and its performance results are compared in different cases. Key Words: 1- Massive MIMO 2- two-tier heterogeneous networks 3- Small cell, 4- BD and ZF precoding methods, 5- Antenna selection 6- Spectral efficiency