Today, with increasing volume and variety of data, the need for automated analysis has become increasingly important. This automatic analysis involves extracting information from existing data and using these analyzes for future applications. These applications can be divided into automatic data classification, clustering and separation of data types, and anomaly detection in data. Detection of anomalies in image is one of the most important issues of computer vision using a variety of types such as public security, health monitoring and intrusion detection. Analysis and detection of anomalies is important because they provide useful information about the attributes and specifications of the data production process. Despite the different applications, anomaly detection is not a well-defined problem. The usual definition of an abnormality is a low probability event and is significantly different from other data. The goal of this problem is estimate distribution of input dataset and, in the future, can find data that has not been generated from this distribution. Therefore, usually an unsupervised learning problem is considered. The thesis focuses on unsupervised learning to detect abnormalities. The proposed approach is based on the autoencoders in conjunction with Generative Adversarial Networks. This method is called Adversarial Autoencoders and is a probability autoencoder that attempts to match the aggregated posterior distribution of an autoencoder’s hidden vector with an arbitrary prior distribution. The proposed method encoder is a capsule network. A capsule is similar to a neuron, except that the input and output of a capsule is a vector instead of a scalar. This design enables the capsule not only learn a specific feature in the input image, but also recognizes the angle and view point and deformation of it. The method of training capsules is a new approach called routing by agreement. Routing by agreement, finds the exit path from the low-level capsule(s) to the corresponding capsule(s) at a higher level. With this approach, there is a similarity between the input and output of a capsule. While, in previous encoder structures, have been used convolutional networks instead of capsule networks. Applying capsule encoder instead of convolutional does not have problems such as the angle and view point images, especially for medical images. The reconstruction error of learned autoencoder for normal events is low and for abnormal events is high. This error in the autoencoder with a capsule encoder had a better performance than an autoencoder with a convolutional encoder can be used to image anomaly detection. Also, the adversarial error of the proposed method compared to the adversarial error of previous methods in anomaly detection is better.The results obtained from the implementation of the proposed structure show that the final performance can be increased up to 10%, which indicates the effect of designing the proposed model.