Nowdays; the issue of slope stability is one of the crucial factors in the economy and safety of open pit mines. Identifying the potential instability and their management in open pit mines is also an important part of safe mining operations. In the present study, using the GIS and the Rock Engineering System (RES), potential instabilities in the final pit 1105 of Chadermulo mine is evaluated. In the Rock engineering system, the parameters affecting the system were identified through field studies, then the binary interaction of these parameters was studied. By determining the coefficients of each of the parameters using their binary interaction, the overall system status was determined quantitatively as a linear relationship between the coefficients and the values of the parameters in different locations. The parameters used include lithology, tectonic properties of the walls, resistance properties, weathering condition and average value of the RQD of the mine area. The values of the parameters for each rock unit were determined using weighted averaging. In the present research, the preparation and processing of information for the rock engineering system in order to identify the instabilities in ArcGIS 10.3 software has been done. By analyzing these data in the GIS, potential instabilities was detected in the eastern, southwest, and northwestern walls. Then, seven sections were selected in this area to conduct the limit equilibrium analysis. All of the limit equilibrium analysis in ArcGIS 10.3 have been done in Python language coding. The Hoek-Brown failure criterion is used to assess the rock mass stability of the study area. The parameters of the Hoek-Brown rock mass failure criterion were evaluated using the Geological Strength Index (GSI) for different rock units in the study area. A genetic algorithm was used to search for a critical slip surface in pit walls. The results showed that there was a probability of slip in the studied sections. The results of the research show that in the final design of the eastern, southwest and northwest walls, the necessary measures for stabilization should be done. In addition, it is suggested that a radar monitoring system be used to manage the risk of sloping slopes in the mine.