The atomic force microscopes (AFM) determine topography of surfaces in nano scale, by means of dynamic response of a micro cantilever beam. Therefore, accurate mathematical modeling and simulation of cantilever dynamics are crucial to design and fabrication of the AFM. Thickness of AFM cantilevers is in order of few microns or sub-microns and producers trend to make it even smaller because of having more resolution and sensitivity. By reducing the cantilever thickness, the size effect may be appeared which can change the AFM dynamics. In this research, the effect of size and surrounding fluid on the dynamic behavior of AFM is investigated based on the non-classic theories. Governing equations of the motion are derived using Hamilton’s principle based on gradient strain and modified couple stress theories for Euler–Bernoulli micro-cantilever beam. Next, the Galerkin projection method is applied to extract the lumped model. By analysis of the amplitude and frequency of free vibration, the necessity of considering the size effect for more reliable estimation of working range is shown. Moreover, the effect of squeeze film damping and hydrodynamic forces of the fluid on the dynamic behavior of AFM is investigated. Comparing the results with previously reported experimental ones in the literature, the model is validated. In addition, accuracy of the solution technique is verified by comparing the results with two degree-of-freedom model and analogue equations method. Also, the stability analysis, frequency response and sensitivity of AFM based on classic and nonc-lassic theories are investigated and the obtained results are compared with each other. In the final phase of the dissertation, using numerical and analytical methods, the chaotic behavior of microscope is studied. For this purpose, the effect of essential parameters such as initial distance between the tip and the surface, amplitude of excitation and damping on the chaotic motion of the system are studied with and without considering the size effect. Furthermore, using maximum exponent of Lyapunov, the working range of microscope with periodic oscillations, is determined as a function of initial distance and amplitude of excitation. Findings of this research can be helpful in the design and characterization of the AFMs parameters and more reliable prediction of its size-dependent dynamic behavior. Keywords: Atomic Force Microscope, Size Effect, Modified Couple Stress Theory, Gradient Strain Theory, Stability Analysis, Nonlinear Dynamic, Frequency Response, Chaos