Supercritical carbon dioxide extraction has received much attention as a new and efficient method. The advantages of this method include low cost, non-flammability, non-toxic and non-corrosive, no solvent remaining in the extract, environmental compatibility, suitable for temperature sensitive materials and high extraction speed. In the present study, Quercetin in Viola odorata in two ways Soxhlet with methanol solvent and extracted by means of supercritical carbon dioxide modified will be carried out. Supercritical extraction was performed by considering 30 minutes of static time and adding 2 mL of ethanol as modifier and particle size with mesh size 20-35. Experiments using the Response Surface Methodology (RSM) in Minitab16 software and considering 4 important parameters of temperature (35-35 ° C), pressure (10-30 MPa), carbon dioxide flow (0.5-1.5 mL / min) and Dynamic time (40-40 min) was designed. Then, analysis second order polynomial procedure call a model laboratory for the recovery Quercetin is fitted. R 2 and modified R 2 of the model are 98.81% and 97.78%, respectively which shows a very good fit of experimental data by this model. Optimal values of model parameters were temperature 55 °C, pressure 20.50 MPa, flow rate 1.17 mL/min, dynamic time 118.38 minutes and the maximum recovery of Quercetin 54.10 percent achieved. With three repetitions of the experiment under optimal conditions, the average recovery was 52.68%. The effect of operational parameters on recovery was also investigated using contour and surface plots. Also, in this study Mathematical modeling process has been done using balance equations of mass transfer model and the results of the model and experimental data were compared with each other and a good agreement was observed between them. In this model, the parameters of mass transfer coefficient film, effective diffusion coefficient and axial dispersion coefficient are calculated by empirical relationships and the distribution coefficient has been calculated through thermodynamic and genetic algorithms, in which the absolute average deviation (AAD) between the model results and the experimental data minimize. Furthermore, optimization was performed by the genetic algorithm optimization parameters, temperature of 55 °C, pressure 21.03 MPa, flow rate 1.19 mL per minute, dynamic time 119.54 minutes and optimal recovery of Quercetin 54.83 percent is obtained. As can be seen, the optimal parameters values optimizes both methods match well with each other. Keywords: Quercetin, Viola odorata, Supercritical fluid extraction, Carbon dioxide, Response surface design, Mathematical modeling, Mass-balance model, Genetic algorithm