Surface inspection of steel sheets is Vital for Steel Industries. On the other hand, visual inspection manually not only is a dangerous task but also has low performance. To improve quality, steel sheets must be classified into defected and defect-free categories. Then, each defect category can be divided in more subclasses of defects. There are a number of different automatic methods for this purpose and we used neural networks in our work. We considered four surface defects including Holes, Scratches, Coil breaks and Rusting. Images of defect free sheets were also included and using neural networks and simple characteristics to categories the images. Parallelism and high adaptability toward environmental changes are among advantages of using neural networks as compared to other identification methods. If a new class of defects is added to the existing classes we can train network again to classify the whole set and therefore, there is no need for a new algorithm to classify the added category. Simulation results show that our approach has high capability in detection and recognition of surface defects. In a manner that the system is always successful in classifying faulted and fault free categories. In addition to high performance the suggested approach due to high parallelism is suitable for real time implementation.