Infertility has been widely investigated in recent years. In average 40% up to 60% of all couples experiencing infertility, a male factor is involved. The first and non-invasive test that checks the quantity and quality of male is assessment of semen or spermogram. Since 15 years ago computer aided systems for sperm analysis (CASA) has been introduced. The CASA system also makes the assessment process more objective and accurate. Sperm’s density, mobility, progressive motion and their morphology features are the most important microscopic characteristics which are used to evaluate male fertility and specifying the sperm cells’ health.World Health Organization has determined these properties for specifying the sperm cells’ health. Sperm mobility assessment from video images is one of the most useable of these features that need accurate tracking process. However, a few researches in this area have been conducted. In this research we first study mathematical background needed to develop object recognition and tracking. One of the main steps is the background subtraction from video images. We choose the difference of consecutive frames as a simple and effective method, however, it suffers from selected threshold as the output depends highly on the threshold. To eliminate its dependence on thresholding, we propose a nonlinear diffusion filtering in time domain. Nonlinear diffusion filtering is a smoothing and noise removing method from images that preserves edges. In tracking step we investigated many tracking methods and ltr" Keywords: Infertility, Spermogram, Point Tracking, Nonlinear Diffusion Filtering.