Transition metals oxides (TMOs), due to the presence of strong Coulomb interactions, are a veritable playground in condensed matter physics for the study of novel phenomena such as high-temperature cuprate superconductivity (SC), colossal magnetoresistance, multiferroics, and different ordered magnetic phases. This class of materials, especially 3d-TMOs, in which the d orbitals are localized, are a ideal systems for observing signatures of Mott insulating behavior. In the strongly correlated limit, the charge degrees of freedom `freeze' and the low energy physics of these insulators can effectively be described by a Hamiltonian with only spin degrees of freedom. The system energy can, according to second order perturbation theory, be reduced by the virtual hopping of electrons to neighboring sites, leading to inter-site spin-spin correlations. The search for new superconducting phases is often undertaken in proximity to the long-range ordered phases, as the spin fluctuations play an important role in the forming of Anderson's resonant valence bond (RVB) states. The RVB ground state is a linear combination of all possible singlet states which can be formed between the spins in a system. When the system is doped by either holes or electrons, the singlet states formed by spin fluctuations become mobile, and the superconducting phase is borne. Kane-Mele proposed a model for describing the quantum spin Hall effect in graphene by adding a mass term to the nearest-neighbor tight-binding Hamiltonian. In this dissertation, we study the classical and quantum phase diagram of the Kane-Mele-Heisenbeg model. The classical diagram of the Kane-Mele-Heisenbeg model is obtained using three complementary methods: Luttinger-Tisza, Variation minimization, and the iterative minimization method. The classical phase diagram includes three phases with long-range ordering and three disorder phases with infinitely degenerate. Employing the linear spin-wave analysis, it is found that the quantum fluctuations select a set of symmetrically equivalent in the degenerate phases. In the following of the first section of dissertation, employing exact diagonalization (ED) in Sz and nearest neighbor valence bond (NNVB) bases, bond and plaquette valence bond mean field theories, We show that the disordered regions are divided into ordered quantum states in the form of plaquette valence bond crystal (PVBC) and staggered dimerized (SD) phases.