Review of vision-based steel surface inspection systems

Steel is the material of choice for a large number and very diverse industrial applications. Surface qualities along with other properties are the most important quality parameters, particularly for flat-rolled steel products. Traditional manual surface inspection procedures are awfully inadequate to ensure guaranteed quality-free surface. To ensure stringent requirements of customers, automated vision-based steel surface inspection techniques have been found to be very effective and popular during the last two decades. Considering its importance, this paper attempts to make the first formal review of state-of-art of vision-based defect detection and classification of steel surfaces as they are produced from steel mills. It is observed that majority of research work has been undertaken for cold steel strip surfaces which is most sensitive to customers' requirements. Work on surface defect detection of hot strips and bars/rods has also shown signs of increase during the last 10 years. The review covers overall aspects of automatic steel surface defect detection and classification systems using vision-based techniques. Attentions have also been drawn to reported success rates along with issues related to real-time operational aspects.

Steel is probably the most important of all metals in terms of its quantum and variety of use. Steel has contributed immensely towards the development of industrial society. In fact, consumption of steel is considered to be one of the yardsticks to judge the developmental status of a country. As per World Steel Association, production of crude steel during 2013 was 1,582 million tons (Mt), which is more than production figure of all other metals put together. Today, there are more than 3,500 grades of steel available out of which trade in flat steel products accounts for about 50%.
An integrated iron and steel making plant produces liquid iron in blast furnace with iron ore, coke, sinter and flux as input. Liquid iron is converted to liquid steel with specified constituent by primary and secondary steel making processes. Liquid steel is continuously cast into slabs and billets. Slabs are of rectangular cross-section with dimension of a typical slab being 1,600-mm-wide, 250-mm-thick and 12,000-mm-long. Billets are normally of square cross-section of about 150 × 150 mm and about 12,000-mm-long. Slabs are subsequently rolled into hot strips and then to cold strips. Billets are rolled into structural of various dimensions. A simplified flow chart of steel making processes is shown in Figure 1.
Importance of surface quality of steel products, particularly that of cold-rolled steel assumed importance since 1980s primarily due to demands from automotive car makers. In course of time, hot strip surface quality, and in recent times, surface quality of structural products like rods/bars have assumed significant importance.
Traditionally, surface quality of flat steel products, which are in coil form, is judged manually by cutting about 30 m of a random coil in a batch and inspected by an expert. Typically, in manual inspection, the inspected surface is about 0.05% of the total steel surface produced. In cold rolling mill complex, operators are sometimes stationed to inspect the finished product online for any defect. However, due to high line speed, fatigue and other adverse factors, inspection process is hardly satisfactory. Thus, the manual inspection process is not sufficient to guarantee defect-free surface of steel products with reasonable degree of confidence and naturally, need for automated surface inspection grew.
In a significant development [1], nine steel companies and three aluminium companies in US started a research project in early 1980s on vision-based steel surface inspection in collaboration with two commercial organisations. A prototype system was built and tested in several steel plants during 1987. At the same time, European companies also started working. Thus, from later half of 1980s, systematic research work on surface inspection of steel products started. Today, vision-based automated surface inspection systems (ASIS) are produced by many reputed companies. Since 2006, an annual International Surface Inspection Summit (ISIS) is organised by a consortium of manufacturers and others. Technology of vision-based automatic inspection of steel products, even though not 100% accurate has matured. This paper attempts to find out the status of development of vision-based ASIS for steel surfaces through review of published literature during the last two and a half decades.

Complexities of steel surface inspection automation
Real-time inspection of steel surfaces faces a number of challenges. The difficulties may be enumerated as follows: Hazardous site. The place for installation of inspection equipment (illumination system, camera and some signal processing equipment), particularly, for hot rolling mills is very hazardous. Presence of high ambient temperature, dust, oil, water droplet and vapour is very common. Additionally, the illumination system and the cameras require protection against shock and vibration. Further, heavy equipment is moved in and out of site during daily, weekly and annual maintenance. All above factors necessitate the use of appropriate physical and environmental protective measures for site equipment.
Operating speed. During regular production, operating speed of the surface to be inspected is generally high. For flat steel products, speed at the end of rolling, where the inspection equipment has to operate, is typically 20 m/s. For long products, particularly wire rods, speed could be as high as 225 miles/h (100 m/s) [2]. Real-time operation at such high speed requires special image processing equipment and software with small execution time.
Varieties of surface defects in different steel products are reported to be very high [3]. For example, Verlag Stahleisen [4] have categorised surface defects of hot-rolled products in nine main classes and 29 subclasses. These defects are not governed by any standard. Thus, their characteristics and classification vary from mill to mill and from operator to operator. Further, manifestation of these defects changes due to variations in production process.
Large number of cameras. For flat steel products, two sets of inspection systems -one for top and another for bottom surface -are needed. Each of these sets in turn generally consists of 3 to 4 cameras to cover the entire width of the strip. For long steel products, multiple cameras are to be located peripherally to ensure coverage of entire surface. For example, for a round product, at least three cameras are used while use of five cameras has been reported in the literature [5]. Thus, gathering of images and their real-time processing is a daunting task.

Prior literature review
Over the years, a number of review papers [6][7][8][9][10][11][12] on various aspects of surface defect detection have been reported. Various aspects and methods for texture analysis have been reviewed in [13,14]. Two comparatively recent review papers are [6,7]. Advances in surface defect detection using texture analysis techniques have been dealt with by Xie [6] covering applications in mainly textiles, tiles and wood. Kumar [7] has covered very comprehensively research work done in fabric surface defect detection and provided some valuable conclusions. Review papers particularly on texture defects and defects in fabrics also mention steel surface as a category where identified techniques can be applied. It is worth mentioning that as early as in 1982, 11 papers were listed under 'Inspection in Metal Processing Industry' category in a review by Chin and Harlow [12]. Gonzalez and Woods [15] provide an excellent theoretical background to all aspects of image processing, whereas theoretical basis for neural network-based classification is adequately covered by Haykins [16]. However, the authors could not locate any review of research work done in the field of steel surface defect detection and classification. Therefore, in this paper, attempt has been made to consolidate the published literature from academia, steel industry and manufacturers on the topic of automatic defect detection and classification of steel surfaces.

Availability of research publications on automated vision-based steel surface inspection
Availability of the published literature on steel surface inspection mostly consists of research work done at various academic institutions, steel plants/steel plant research units and surface inspection equipment manufacturers. A number of research works have been published jointly by academic/research institutes and steel plants indicating good collaborative partnership. During the last 10 years, a significant percentage of published work on steel surface inspection systems came from China. This is commensurate with China's dominant presence in steel manufacturing.
Some papers have been published with reported research work mainly on defect classification aspects implemented in commercially procured systems. While overall systems and their benefits are well documented by reputed manufacturers, details of defect detection and classification techniques are not elaborated, probably due to issues regarding intellectual property rights.

Categories of steel surfaces
Types of steel surfaces studied for defect detection/classification are: slab, billet, plate, hot strip, cold strip, rod/bar. They cover a large proportion of applications of steel as a material. cold strips, and off late, rod/bars have received more attention of researchers. This is mainly explained by the fact that large proportions of these products are finished product and quality requirements of customers have become more stringent over time.
Broadly, steel surfaces can be categorised in flat and long products ( Figure 2).
Flat product surfaces can further be classified as follows: -Slab/billet: both are produced by continuous casting process from liquid steel and have some similarity with respect to surface and internal conditions. Surface is scale covered and more grainy. -Plates are produced by reheating a slab at about 1,250°C and rolled subsequently. The surface is oxidised and comparatively even with respect to that of slab. -Hot strips are produced by reheating a slab at about 1,250°C and rolling in multiple rolling stands to reduce the thickness to desired value. The strip surface is oxidised. However, due to high rolling force, the surface granularity of hot strip is considerably reduced compared to slab. -Cold strips are produced by rolling hot strips in cold rolling mill after pickling process (which removes the oxide layer and cleans the surface). Thus, the surface of cold strips is not oxidised, and the surface is quite smooth due to very high rolling forces used in cold deformation process. -Coated strip (galvanised, tinned)/finished stainless strip surfaces are highly reflective in nature.
Long product surfaces can further be classified as follows: Rods/bars are produced from billet by hot rolling process, and their surface is fairly oxidised. Further, the surface is also not flat, and therefore, angle of reflection varies towards the periphery thus producing nonuniform image intensity.
Other long products like angles, channels, heavy structural, rails etc. are produced from billet/bloom. They are of complex cross-section and require special lighting and camera arrangements.

List of surface defects for steel products
There is a large variety of surface defects for different steel products. Further, there is no agreed standard for defects. There is also large 'inter group similarity and intra group diversity' [17] for various classes of defects, which makes defect classification difficult. Defect catalogues published by Verlag Stahleisen GmbH, Germany [4] act as defacto standards for this purpose.
An attempt has been made to list some of the main defects which have been referred in the literature for surface defect detection and classification during the last two and a half decades. Defects have been listed vis-à-vis the categories of steel surfaces mentioned above.
7. Key elements of automatic surface inspection system hardware structure Figure 3 shows the basic hardware structure of ASIS. It consists of one or more light source, one or more camera (bright field or both bright and dark field), fast image processor, server and the operator interface.

Image acquisition
To obtain satisfactory surface image quality, it is important to illuminate the surface adequately and uniformly. In fact, high quality of illumination reduces computational burden of image processing. Two types of illumination techniques can be used for metallic surfaces: intensity imaging and range imaging. [18][19][20][21][22] have discussed various aspects of illumination systems for metallic surfaces. Research on imaging systems for cold strips has been well documented in [23].
Range imaging provides height information thereby making 3D defects prominent. Range imaging is not competitive to intensity imaging. In general, use of range imaging is not common in steel surface defect studies.
Intensity imaging is primarily of two types: bright field and dark field. In bright field illumination, the sensor captures most of the directly reflected light. The surface appears bright, whereas the defect features appear darker. In dark field illumination, the angle of the incident light rays to the surface normal vector is very large. This results in a dark appearance of the surface, but some defects appear bright in the image. Dark field view requires more intense lighting. Requirement of about eight times compared to bright field lighting has been reported [21].
Unfortunately, all surface defects do not show up either in bright field or in dark field alone. There are many examples of the use of two sets of cameras covering both the fields of view [24][25][26]. Use of 20 charge-coupled device (CCD) area scan cameras which are used to capture surface image of both sides of hot-rolled strips using both bright field and dark field modes have been reported in an iron and steel plant of China [24]. However, considering maintenance issues and system complexity, most of the systems place the cameras in between the bright field and dark field locations.

Source of light
The light source is required to provide uniform ripple-free light as far as possible. While ripple-free illumination calls for special arrangement of light power supply [27], providing uniform intensity is not possible due to the use of more than one light source in majority of the cases. Figure 4 shows the variation of incident light intensity on to a steel surface using two xeon lights [28]. Types of light source which are used in general are: wide spectrum tungsten, fluorescent tubes, halogen, xeon and LED.

Type of camera
In general, high-resolution CCD cameras are used. Use of both line scan and area scan cameras has been reported in the literature. Line scan cameras have been widely used as it is easier to realise a strong and even illumination to the surface area to be inspected. The disadvantage with the line scan cameras is that they do not generate a complete image at once and requires an external hardware to build up images from multiple line scans [7]. Most of the automatic surface inspection system manufacturers use line scan camera. For area scan cameras, the usage of transport encoder is optional and the inspection resolution in both directions is independent of the object (web) speed. However, while using area scan camera, special attention is needed to ensure even illumination of the total area under scan to the extent possible. High-resolution video cameras are also used as complimentary systems [30].
Image resolution. Various dimensions of image resolutions have been reported [24,26,31,33,34]. Cross web resolutions vary from 0.17 mm to about 1 mm while reported down-web resolutions vary from 0.25 to 1.25 mm.

Image processing computer hardware
Images captured by a CCD camera are transferred to some form of fast, parallel processing system dedicated to the camera and located close to it [24]. The parallel processing system ensures real-time operation by processing bulk image data and selecting and storing regions of interest (RoIs). The parallel processing system could be a part of the camera itself, or a FPGA processor or a general purpose processor with special hardware. This part of the system is vitally important both from real-time operation as well as accuracy of defect detection and classification. Thereafter, a server with a large backup memory is used for further processing and for operator's interface.

List of defect detection and classification methods
Various methods/techniques used for defect detection and classification of steel surfaces are listed in the literature. Table 1 shows the list of different methods of defect detection vis-à-vis references obtained for this study. Types of steel surfaces have also been mentioned in the table. Techniques followed may broadly be categorised as statistical, morphological, spatial domain filtering, frequency domain analysis, joint spatial/spatialfrequency analysis and fractal models. Spatial domain filtering, morphological operations and joint spatial/frequency domain filtering are found to be used extensively for all types of surfaces. Ultimate objective of surface inspection is to categorise defects in specified classes using classification techniques. As a process, classification starts after defects are localised by segmentation. At this stage, generally a number of features are extracted from regions of interest. Ideally, different combinations of these features are required to match uniquely with that of different types of defects. The matching is normally done using adaptive learning methods such as neural network with back propagation (NN-BP), support vector machine (SVM) etc. Adaptive learning is of two types: a) supervised where the network is provided with a large number of known inputs. Thereafter, the network produces the known outputs as closely as possible based on training. b) In unsupervised learning, the network is required to work out relationships between various inputs without being told.
However, steel surface defects exhibit large 'inter group similarity and intra group diversity '. Thus, finding suitable features and identifying classifiers with low computational cost are the major areas of research activity. Table 2 shows the list of classification methods with respect to references and types of surface.

Comparative evaluation of defect detection systems
In Table 3, some of the typical vision-based defect detection systems presented in the literature are highlighted. Attention is drawn to broad methods followed, types of defects, sample size and resolution of images used for study and reported detection accuracy. Speed of steel object and reported suitability for real-time operation are also mentioned. In a number of studies, detection of a single defect is achieved after elimination of pseudo defects using a classifier ( [29,38,67] etc.). These are shown here instead of under classification table (Table 4).
Discussions on defect detection methods. Defect detection and classification in steel surfaces broadly follow three steps: Localisation of candidate defects/regions of interest (RoIs) by means of segmentation, extraction of features from RoIs and finally, classification into defects and pseudo defects. In a large number of instances [3,35,41,42,56,58,61], it has been possible to localise RoIs with a high degree of accuracy. This is particularly true when occurrence of pseudo defects is very limited as can be observed for many hotand cold-rolled strips. In such cases, detection itself serves a very useful purpose of identifying a particular product as defective in a continuous production line. Appropriate action of segregating the defective product follows naturally. However, without defect classification, corrective actions cannot be taken to stop occurrence of defects in future. Defect detection loses its significances without classification in situations where pseudo defect percentage is significant and/or variable. This is mostly the case with slab, billet and rod/bar. Here, vision-based inspection systems are useful only when features are extracted from RoIs, and classification is achieved between defects and pseudo defects and then between the defects themselves.

Pre-processing
Images of steel surfaces contain a lot of noise, and preprocessing in most papers is devoted to noise reduction and highlighting defective region from background. Noise removal is more important in slab/billet/rod surfaces where visual characteristics of defects (crack, seam) are similar to that of pseudo defects (scale etc.). Averaging filter [58,61] and Gaussian filter [59] are used routinely before processing for defect detection. Interestingly, a Wiener filter, which requires processing in frequency domain with information on power spectrum of noise and un-degraded image, has been used in a cold-rolled strip application [76] with realtime suitability. [25,43] used homographic filtering to reduce effect of variable illumination.
An important aspect of pre-processing in rod/bar is to remove non-steel background from captured images. Maximum diameter of rod/bars reported in this study is about 46 mm. A camera normally covers more width to allow for transverse movement of the bar (during rolling) thus capturing useless background information. At the beginning of image processing, bar image is recovered from the background for better outcome of defect segmentation [5,44,58,59]. This way memory utilisation is also more efficient.
Surface flatness is very important due to its effect on methods and processes followed for RoI identification. Surface of slab, hot and cold strips is flat and wide as against the surface of rod/bar which is circular in shape. For uniform defect-free surfaces, intensities of reflected light from a surface of flat strip are more or less uniform.
However, this is not the case with circular surface of rod/bar as can be seen from uneven intensity plot in [61] where the central region of the field of view is far brighter than the peripheral region. This problem is reduced to some extent by using more number of cameras than strictly required as in [5,44] (five cameras) and [85] (four cameras). Uneven intensity leads to varying response to thresholding as well as gradient-based filtering. Thus, spatial domain methods become more involved while identifying RoIs.

Spatial domain-based methods
For defect detection/identification of RoI, gray level-based spatial domain techniques (gradient filter, thresholding, local contrast etc.) and joint spatial/frequency-based techniques (various types of wavelets) have been used. Spatial domain techniques are fast and easy to implement compared to wavelet-based methods, and they have been used in most of the papers where suitability for real-time operation have also been reported [2,24,26,35,46,58,59,61,76]. First-order gradient filter followed by thresholding is more commonly used for edge localisation [2,44,46,59,76] due to its noise immunity characteristics compared to Laplacian. Region growing has been employed [2,24,59,76], particularly after the use of gradient operators, to identify RoIs.
For detection of six types of defects in cold strip, [76] first used a Wiener filter for noise reduction. Then, rough locations of defects were extracted using Sobel edge detector. Finally, object locations were identified by a specially developed region growing method based on fuzzy logic object membership function of a pixel. However, in [2], the use of horizontal gradient operator for detection of vertical edge of seam defects also simultaneously detected equal number of pseudo defects. Based on seam defect characteristics being very thin and darker than background, seed points were identified and subsequent region growing step provided RoIs.
Yun et al. [44]: To detect vertical scratch in rods, firstorder horizontal gradient filter was first used followed by an edge pair search. Then, morphological dilation and erosion was used to join edge pair. Thereafter, defect size (length) less than a value was rejected to discard spurious defects.
Laplacian gradient operator was successfully used by [61]. First, the original image was smoothened by using weighted averaging filter. Then, Laplacian is first applied in x-axis followed by two-level threshold. Subsequently, Laplacian was applied in y-axis with a single threshold. Threshold values were dynamically updated. Good defect detection ratio (>95%) was obtained for three types of defects in rods with only 8.331 ms processing time per image.
Background difference method has also found favour with researchers [24,31,56]. In this method, it is assumed that defect-free surface is more or less homogeneous and slowly varying. A background (or a standard image) is generated from several consecutive images satisfying some criterion of intensity variation at pixel level or by averaging. Significant difference between an incoming image and background is used to get RoIs. In [24], seed pixels for four types of defects in hot strip surfaces were identified by applying one or more threshold values to differential gray     levels of background and an incoming image. Region growing method is then used for obtaining RoI. Cong et al. [56] partitioned the difference of image in small (20 × 20) subimages. Five types of defects in cold strips were identified by characterising gray-level distribution in subimages.
Traditionally, Hough transform [3,43] is very successful in demarcating well-defined shapes. Martins et al. [43] used Hough transform to detect three geometrically well-defined shapes: welding, clamp and identification hole in cold-rolled strips. After homographic filtering to remove illumination variation, Gaussian and morphological filters were applied to reduce noise. Segmentation was realised by applying edge detection method and thresholding. Hough transform was performed on the binary image for line and circle detection. Three types of defects-welding (diagonal line), clamp (multiple lines in down-web direction) and identification hole (circle with fixed radius in the middle of the strip)-were identified.
Local entropy has been used for defect identification purpose in different ways in [25,31]. In [31], enhanced quality of cold-rolled strip image was first obtained after subtracting generated background image. Local entropy of each pixel was calculated using a 9× 9 matrix and then thresholding done on the resultant image by Otsu's method. In order to eliminate nonrelated regions, morphological dilation and erosion operations were done on binary image. As against entropy which is a measure of randomness, excess entropy is a measure of the structure of the system and was introduced in [25] as the measure of image's spatial structure. After homographic filtering, excess entropy was calculated and threshold value for RoI segmentation was arrived at after finding optimal parameters of fuzzy membership function (between background class and object class) for maximum excess entropy.
Li et al. [58] developed a local annular contrast(LAC) algorithm for RoI identification based on the fact that pixels belonging to defect have a greater contrast with respect to surrounding background and used it to detect three types of defects in rod. A gray-level comparison between the detection point and its surrounding pixels at a certain distance was undertaken, and the threshold value for segmentation was changed with the gray-level variation of local annular background. As radius of LAC is required to be greater than the size of defect, constraint is likely to be encountered with respect to the size of a defect to be detected.

Wavelet-based methods
Wavelet-based methods have the advantage of better performance against noise compared to spatial domain-based methods. Further, wavelets can localise defects in spatial and frequency domain unlike purely frequency domain methods. Most of the applications of wavelet-based methods under survey are for defects which are difficult to be identified from pseudo defects like scale and other artefacts which are frequently observed in surfaces of slab, billet, plate and rod. [5,38,41,65,67,69] are some of the studies in this direction. [41,69] used undecimated wavelet transform followed by thresholding of correlation coefficients to arrive at identification of defects. Yun et al. [69] used Haar wavelet with onelevel decomposition to detect two types of defects in billets with 83% accuracy where as [41] used biorthogonal wavelet with three-level decomposition followed by binarization, morphological operation and radon transform for plates with accuracy of 90%.
In [5], vertical detail coefficients of Haar undecimated wavelet transform were thresholded to identify RoIs. After obtaining 14 features from binary and gray images, defects are classified from pseudo defects by using SVM classifier with about 92% accuracy in rods.
Jeon et al. [38] used discrete wavelet transform to isolate defects from scales and variation in lighting conditions. To generate candidate defects, negative values were thresholded in wavelet-reconstructed image. Four morphological features of binary image were used to segregate corner crack defects in billets with more than 97% accuracy.
Real and imaginary parts of 2D Gabor filter can be used as a blob detector and edge detector, respectively. [65,67] respectively used real and imaginary part of Gabor filter followed by two-level adaptive thresholding. Candidate regions of scratch defects were selected by edge pair detection in [67]. Further, to reduce influence of pseudo defects, SVM was used with 12 geometric and gray-level features in [65] and seven histogram and gradient-based features in [67] as inputs to detect one type of defect in plate and slab with 85% and 94% accuracy, respectively.
Lee et al. [72] (cold strip application) and [70] (hot strip application) used wavelet-based methods to classify 8 types and 24 types of defects, respectively. Lee et al. [72] used an adaptive wavelet (AWP) packet scheme to produce the optimum number of features through the subband coding gain. Four entropy features in the images and the energy features from the optimal quadtree in the AWP algorithm were extracted. Three-layer NN-BP of ten hidden nodes and eight output nodes (eight types of defects) was use with standard back propagation algorithm for classification with very high accuracy of 99%.
In an elaborate study [70], five different types of wavelets, namely, Haar, Daubechies2 (DB2), Daubechies 4 (DB4), biorthogonal spline (bior) and multiwavelet, were used to extract features by decomposing the small blocks (32 × 32) of surface images into different resolution levels. The defect detection ability of these features was assessed by two kernel classifiers, namely, the support vector machine (SVM) classifier and the vector-valued regularized kernel function approximation (VVRKFA) method of classification. The central outcome of this paper, based on test results on 24 types of defect classes, was that the wavelet feature sets were better suited for steel surface defect detection application compared to texture-based segmentation or thresholding techniques. Further, three-level Haar feature set is more promising compared to the Daubechies, bior and multiwavelet features. With VVRKFA classification accuracy of 94% was achieved.

Use of fractal model
FRACTAL geometry is the geometry associated with naturally occurring objects that have repeating patterns at different scales. A fractal geometric approach to computer vision is important in the interpretation and recognition of objects that are characterised by their texture and therefore difficult to interpret using conventional machine vision techniques [76].
Yazdchi et al. [63] evaluated multifractal dimension based on morphology to determine the position of defects. By eliminating details whose scale is less than that of the structuring element (SE), an image can be observed and measured at different scales. In this multifractal estimation method, a series of SEs of different scales were used to measure the image surface. A set of multifractal texture descriptors, namely, the local morphological multifractal exponents (LMME), were defined. For every pixel, the multifractal features were computed in a slipping window of size 11 × 11 centred on that point. The segmentation was based on the fuzzy C-means (FCM) algorithm, which was able to classify the pixels into a specified number of regions by clustering those features. After multifractalbased segmentation, morphological erosion and dilation operations were done to remove spurious regions. Classification of five types of defects in cold strips was achieved with about 98% accuracy.
Blackledge et al. [76] used a combination of both Euclidean and fractal geometric measures to arrive at feature vector. The approach was focused on multiple object location and classification using fractal geometry to evaluate the texture as a set of features from an image which was already segmented using gradient operator and a novel region growing method. Subsequently, the system used a fuzzy logic-based approach to classify six types of defect. For classification of six types of defects in cold strips, 85% to 95% of accuracy was claimed.

Comparative evaluation of defect classification systems
In Table 4, some of the typical vision-based defect classification systems are highlighted. Attention is drawn to broad methods followed, types of defects, sample size and resolution of images used for study and reported classification accuracy. Speed of steel objects and reported suitability for real-time operation are also mentioned.
Discussions on defect classification methods. Classification of defects in steel surfaces is important for identifying and subsequently correcting causative factors. It is also important from the point of view of product pricing. Geometric defects are mainly caused by abnormalities in rollers and in rolling mills where as texture defects generally are of metallurgical origin [70]. Therefore, nature of a particular defect may lead to corrective action in diverse locations of steel making production system. Most commonly used classification methods for steel surfaces are: NN-BP and SVM. Other supervised methods used for classification purposes are K-nearest neighbour, fuzzy logic and discriminant function-based methods. Among unsupervised classifiers, self-organising map (SOM) and learning vector quantiser (LVQ) have also been used.

Neural network with back propagation (NN-BP)
Steel surfaces exhibit many types of defects, and NN-BP has been extensively used to classify such defects [31,36,54,59,63,72,78,80]. Generally, NN-BP has been used with one hidden layer. Large variation of number of nodes used in input layer (feature space: 4 to more than 50), hidden layer (10 to 100) and output layer (defect class: 2 to 24) has been reported. Most of the applications of NN-BP have been found to be for hot-and cold-rolled strip surfaces presumably because of manifestation of large number of important defect classes. Performance of NN-BP has been compared with other classifiers in [31,36], and NN-BP has been found to be performing better.
Caleb and Steuer [36] is an interesting paper where experiments were conducted on relative performance of multilayer perceptron (MLP) and group method of data handling (GMDH) with same set of five classes of defects and non-defect in hot strips. Classifiers were constructed to segregate i) defects as a class from nondefects, ii) all six classes and iii) five classes of defects within themselves. MLP consistently provided better result than GMDH. Within three sets of trials, best result from MLP was obtained (97%/93% for training/test set) during classification of defects from non-defects. For classification between five defect classes, accuracy reduced to 89% and 78%, respectively, for training and test sets and further to 86% and 80% for all six classes. It is suggested that simple problem decomposition on a class level would involve cascading two networks, the first network separating regions on defects/non-defect basis and the second only dealing with the cases classified as defects by the first.
In [31], two methods of classification, NN-BP and FIS, were tried for five types of defects in cold-rolled strip. For fuzzy inference system (FIS), fuzzy C-mean algorithm was tried using a cost function. Accuracy obtained using FIS was much lower at 82% compared to NN-BP (97%) using seven statistical features. Peng and Zhang [77] proposed a back propagation (BP) algorithm based on modified error function to overcome a major problem of long training time of the BP algorithm and chance of falling into local minima. It was noted that the delay of the convergence was caused by the derivative of the activation function. A slightly modified error function of the back propagation algorithm resolved this shortcoming and accelerated convergence to a solution. For five typical defects from cold-rolled strip samples, improved algorithm was effective at getting rid of local minima problem with less time (30 epochs as against 100) with an average success rate of about 94%.
Masci et al. [80]: A convolutional neural network(CNN) is a type of feed-forward artificial neural network where the individual neurons respond to overlapping regions in the visual field. In CNN, convolutional layer performs a 2D filtering between input images, x and a bank of filters, w producing another set of images, h. A nonlinear activation function is applied to h just as for standard multilayer networks. Pooling layer reduces the dimensionality of the input by a constant factor and also undertakes feature selection. The input images are tiled in nonoverlapping subregions from which only one output value (max. or avg.) is extracted. Subsequent, fully connected layer performs a linear combination of the input vector with a weight matrix. Max-pooling convolutional neural networks (MPCNNs) perform feature extraction and classification jointly. With 7% error rate, MPCNN performed much better than SVM for seven defects in cold strips. In CNN, the number of free parameters does not grow proportionally with the input dimensions and therefore performs better in terms of many benchmarks.

Support vector machine (SVM)
Classically, SVM was designed to separate two classes. Thus, it has been widely used for separation of defects from pseudo defects in steel surfaces [2,5,44,47,65,67]. SVM has also been extended to solve multiclass separation problem using mainly one-versus-all and one-versus-one techniques. Multiclass problem solution using SVM has been reported in [25,70,81,[83][84][85]. To apply multiclass classification problems, a few binary classifiers are required to be trained. The most common approach is the one-versus-all strategy where a classifier is trained as positive label for one class and negative label for all other classes. This strategy requires n binary classifiers for n defect classes. In [81], four types of defects in strips were classified using a voting strategy following a series of binary classifiers.
Gaussian radial basis function (RBF) kernel function was found to be routinely used for steel surface defect classification [2,83,85]. In [2], defect detection was modelled as a nonlinear classification problem using a SVM with RBF as the kernel function. The result from the SVM learning was a SVM model that includes kernel parameters γ and C, support vectors, coefficients of the support vectors, the target values of each support vector and the bias. These values were arrived at using incremental SVM learning algorithm. Subsequently, accuracy from SVM classifier was found to be better than NN-BP classifier (94.4% against 90.8%) for a 306 test dataset for seam defect against pseudo defect for rod surface.
Choi et al. [83]: 46 geometric features and 8 gray-level features were extracted from segmented RoIs. Among commonly used kernels such as linear, polynomial, RBF and sigmoid, RBF was selected. Grid-search method was used to arrive at RBF parameters (C and γ). Six defect types from cold strips were correctly classified with 87% to 94% accuracy.
Agarwal et al. [85]: The aim of this study was to integrate the process knowledge residing with the plant engineers with the image-based automatic inspection techniques and to come up with a classification scheme. Three important characteristics of the defects -their shape (length to width ratio), longitudinal and transverse locations -were used for this purpose. Another variable directly used from the camera was the severity of the defect (scale 0 to 100). Based on these features, a process knowledge-based multiclass SVM (PK-MSVM) was developed for rods. PK-MSVM was shown to perform better than other multiple SVM algorithms (one against one, one against all and Hastie's algorithm). Further, Hastie's algorithm was found to perform better than one against one or one against all approach.

Unsupervised classifier
Use of unsupervised classifiers was also reported in several research works. In an unsupervised training, the network learns to form their own classifications of the training data without external help. It is assumed that class membership is broadly defined by the input patterns sharing common features and that the network will be able to identify those features across the range of input patterns. While SOM was used in [34,36,43], LVQ was used in [62,89]. SOMs use unsupervised learning to produce a low-dimensional (typically two-dimensional) discrete representation of the input space of the training samples. SOMs use a neighbourhood function to preserve the topological properties of the input space.
In [36], the aim of the experiments conducted using the SOM network were 1) to explore the subgroups present within each of the classes and 2) to re-analyse the defects misclassified by other supervised learning methods by associating them to the clusters formed on the SOM for six types of defects in hot strip. There was a clear separation of regions which were labelled as not being defects and those that were. There was also separation between the different types of defects. In addition, subclusters within the labelled classes could also be seen. Groupings formed on the SOM consist of visually similar objects irrespective of the labelling. As a result, some of the apparent overlap of classes can be attributed to an ambiguous manifestation of the defect.
In [43], three defects of cold strips with complex geometrical shape were studied using SOM network. Principal component analysis (PCA) technique was used to extract the six main components as features. Output map of 20× 20 rectangular shape was defined for SOM network. The system using PCA and SOM achieved an overall classification rate of 77% which requires improvement. [34] experimented with SOM to classify cloudiness in hot-dip galvanised sheets with about 97% accuracy.

Algorithm processing time
Steel manufacturing is a batch process. For example, during rolling of a slab into hot strip, total rolling process through multiple rolling stands, accelerated cooling and coiling are to be completed irrespective of whether there is any defect present in the strip or not. Defect detection and classification algorithm processing time can broadly be divided in two parts: a) time taken by the system from capturing an image to identification of RoI and b) feature extraction and classification of defect type from identified RoI. The first part is considered to be a 'Real-time' process meaning that for each image, this part of the processing must be completed within a stipulated time depending on the maximum speed at which the steel object can travel. Once RoIs are identified, second part of processing for the whole batch is required to be completed within the stipulated batch process completion time. Thus, the second part of the algorithm processing time is considered to be 'Just in time' process [57].
In Table 5, details of algorithm processing time provided by some of the reviewed papers are given. It is observed that there is very large variation of speed of steel objects (2 to 100 m/s). Generally, very high speed of about 100 m/s is observed in rolling of 5.5-mm-diameter wire rods. Speed of around 20 m/s is observed for hot strip and cold strip processing while speeds from 4 m/s to a maximum of 20 m/s are required for bars. From Tables 3 and 4, it is noted that pixel resolution varies from 0.25 to 1.25 mm/ pixel along the web. If we consider a pixel resolution of say 0.5 mm along the web and an image frame of 500 pixels in the same direction, then for a speed of 20 m/s, we have to process a minimum of 80 image frames per second (12.5 ms/image) to achieve real-time operation. Thus, development of real-time defect detection systems for thin sections of wire rods is the most challenging. Even though speed of slabs and billets are the lowest, their surface contains heavy scale which creates serious bottleneck in defect detection process. Thus, algorithms tend to be complicated for such applications.

Comparative studies
There are a few comparative studies, presumably, on identical data sets. Caleb and Steuer [36] have experimented with NN-BP and the SOM as an unsupervised learning system for classification of five types of defects. Details have been reported under Sections 10.1 and 10.3.
Yichi et al. [52]: In an interesting experiment, six (Sobel, Prewitt, Kirsch, Canny, Laplacian and Laplacian of Gaussian (LoG)) gradient operators in spatial domain, using MATLAB, have been tried and compared for a set of images of cold-rolled strips. LoG with sigma =2 provided better segmentation result.
Yunhui et al. [82]: To study the problem of achieving classification accuracy at the cost of complexity of algorithm, a cascaded weak classifier composed of several weak classifiers (based on extended Haar rectangle features) has been compared with KNN, NN-BP and SVM for cold-rolled strips. With each weak classifier filtering out the most negative samples, a large number of negative samples are eliminated after T-classifiers by a small amount of computation. Superiority of weak classifier (best overall result 94%) is documented for six types of defects in cold strips.
Yazdchi et al. [63] and Yazdchi et al. [31]: For the same set of five defects in cold-rolled strips, two different approaches were used in two papers by Yazdchi et. al. In [63], pre-processing for background noise removal consists of temporal discrete Fourier transform (DFT), high pass filter and inverse DFT. Thereafter, multifractal based on morphology has been used to identify defective segments. Ten features have been used for classification using three-layer NN-BP. Ninety-eight percent accuracy in classification has been achieved. In [31], background image has been obtained using median value of pixels of a set of randomly selected images. To identify defect in difference images, the local entropy of the pixels is calculated using 9 × 9 matrix and an adaptive threshold. Thereafter, morphological erosion, thresholding and dilation identify defect areas. With nine features, feedforward neural network is used for classification with accuracy of 97%.

Steel surface automated inspection system: overview of commercial developments
As technicalities of defect detection and classification of commercially developed surface inspection systems are not disclosed, a brief overview of image processing based commercially developed systems for surface defect detection of web material is presented here.
During the International Surface Inspection Summit (ISIS12, India), various manufacturers of surface inspection systems showcased their products. It is noted that surface inspection systems are commercially available from 1970s (ABB HDI800 Surface imaging-3BFI 802000 R0301/03.02.2010). Since 1990s, commercially available systems were being used for steel surface inspection in good numbers particularly in cold rolling mills. Today, there are many reputed manufacturers of surface inspection systems notably ISRA-Parsytec, ABB, Siemens, Cognex among others.
Key hardware of the systems involves high-resolution (1 K, 2 K, 4 K) CCD camera; extensive, programmable logic devices/arrays for fast pre-processing of camera signals to identify defect candidates in real-time; trainable classification systems based on NN or SVM, user friendly man-machine interface (MMI) and extensive archival facility. All the systems have facility for good connectivity to plant-wide network. Usefulness and application of a commercial installation are elaborated in [90]. Technical details of a commercial system for use in high-speed bar rolling are given in [91,92]. Issues related to appropriate use of ASIS, maintenance issues, expectations of users and how they can be addressed are well explained in a paper by Tivolle et al. [93]. Yang et al. [94] reviewed two commercially developed systems and identified issues to be addressed to make automated visual inspection systems better.
Using alternate detection technology, energy saving of 5,800,000 MMBTU/year by the year 2020 has been envisaged in [95].

Conclusions
This paper dealt with review of automated inspection methods for steel surfaces using image processing techniques. Review of publications over two and a half decades has provided an idea of recent advances that have taken place in this field. Main observations are as follows: a) Due to harsh environment of a steel mill, special attention is required for design of illumination and imaging systems. Steel surface images are reported to contain large amount of noise due to surface scale, vibration, improper/variable illumination, presence of pseudo defects etc. Surface defects are of irregular shape and their type and characteristics vary significantly from one mill to another. Characteristics of defects are also dependent on conditions of manufacturing. b) Published literatures indicate that relatively more importance has been given to detection of defects for cold strip surfaces. Recently, attention is also focussed on surfaces of hot strips and bars/rods. A large variety of techniques, both in spatial and frequency domains, have been applied for defect detection. Often, combination of several techniques has provided useful results. With respect to defect classification, some form of neural network or support vector machine-based techniques have been found to be of use. Real-time operation of automated inspection system often demands very fast processing of images as mill speed is generally very high for flat and long steel products. This calls for dedicated hardware system with parallel processing capability for each camera. c) It is not prudent to compare outcome of different techniques due to lack of common standard with respect to images and experimental methods. This problem is further complicated due to lack of standard definition of defect types. d) Commercially produced automated vision-based inspection systems for web materials have reached a high level of maturity. However, they are required to be properly 'tuned' for a particular application. Also, continuous collaboration between designer and user is necessary to adapt the installed system to new varieties/characteristics of defects at the same installation site.