As a herb, Peganum harmala has been important in traditional medicine for a long time. This plant contains important alkaloids, such as Harmaline, Harmine, Harmane. various concentrations of the alkaloids are effective in treating various diseases, including inhibition of cancer cells. Therefore, developing an efficient and economic method for extraction the alkaloids will pose a great importance for both medical and economical aspects. Extraction of plant materials can be done by various conventional and nonconventional extraction procedures.One ofnonconventional techniques is Supercritical extraction. SFE technique is superior to the other techniques because of reduced use of harmful organic solvents, shorter extraction times, and high diffusion rates of lipids. Methods:In this study, using response surface methodology (in MiniTab16 software) number of 31 tests were designed to extract Harmine from Peganum harmala seed with supercritical carbon dioxide with a modifier (ethanol).In these tests, the effect of four operating parameters (temperature, pressure, CO2 flow rate and dynamic time) with operating ranges (35-75 ° C), (10-30 MPa), (0.3-1.9 ml / min) and (40-120 minutes), respectively on the rate of harmine recovery was investigated. Also, in addition to supercritical extraction, Soxhlet extraction (with methanol solvent) was performed. In addition, the extraction process is modeled using a mathematical model. Mathematical model parameters are including effective pore diffusivity, film mass transfer coefficient, axial dispersion(were obtained from empirical equations), and distribution coefficient (were obtained from genetic algorithm) and the distribution coefficient 131 has been determined by applying the genetic algorithm to minimize the average absolute deviation (AAD) between the data obtained from experiments and model equations. Results: Response surface analysis verified that the data were adequately fitted to second-order polynomial model (?? 2 = 98.09% and modified ?? 2 = 96.42% of this model).The results show that the linear terms of temperature, pressure, CO2 flow rate and dynamic time (?? ? 0.001) and quadratic terms of CO2 flow rate and temperature (?? ? 0.001), also quadratic term of dynamic time (?? 0.05) had significant effects on the model response (Harmine recovery). Also, the optimal values of extraction parameters in the experimental range were 61.66 °C, 30 MPa, 1.22 ml/min flow rate and 120 min extraction time, under which the recovery rate of Harmine was equal to 52.30% is predicted. The results of mathematical modeling showed that this model is well able to predict the experimental data with acceptable accuracy and AAD=3.26 %. Also, in this study optimization was performed by the genetic algorithm. optimization parameters , temperature of 61.825? , pressure 30 MPa , flow rate 1.226 ml/min , dynamic time 119.98 min and optimal recovery of Harmine 52.302% is obtained. Therefore, the optimal parameters values of the genetic algorithm and the response surface design are well matched together.