Multidirectional Scratch Detection and Restoration in Digitized Old Images
© E. Ardizzone et al. 2010
Received: 26 April 2010
Accepted: 7 October 2010
Published: 13 October 2010
Line scratches are common defects in old archived videos, but similar imperfections may occur in printed images, in most cases by reason of improper handling or inaccurate preservation of the support. Once an image is digitized, its defects become part of that image. Many state-of-the-art papers deal with long, thin, vertical lines in old movie frames, by exploiting both spatial and temporal information. In this paper we aim to face with a more challenging and general problem: the analysis of line scratches in still images, regardless of their orientation, color, and shape. We present a detection/restoration method to process this defect.
Scratches are typical damages of old movie films and occur as dark or bright vertical lines which run all over the frames of a video. They are typically caused by the lost of the emulsion of the film surface, due to contact with mechanical parts of film projector or other devices in the film development process. In our work we focused on defects of old damaged photos. Old photographic prints may present several types of defects, caused by inaccurate handling and/or store of the original image, or by chemical factors, or by decomposition of the support. If the knowledge of a degradation origin is essential for defect analysis on the physical support, different defects may look similar once the document is digitized and could be described and removed by similar underlying processes. Several works rely on the damage analysis and restoration of digitized pictures: cracks and craquelures [1, 2], water blotches and foxing , and fading . For a complete taxonomy of the defects in old photos see .
2. Scratches in Videos
The restoration of old damaged videos had been widely studied in the last two decades. When digitized, movie films can be processed by digital postprocessing techniques in order to reduce or remove unwanted artifacts (scratches, spots, etc.). A typical restoration process is made of two steps: detection and restoration.
Most of the approaches use both spatial (from the current frame) and temporal (from adjacent frames) information to detect the presence of line scratches into the video.
All the state-of-the-art methods for videos dealt with a very simple kind of defect: long vertical dark (or bright) scratches, with constant color, width and orientation. Although some authors used a spatiotemporal point of view to process scratches [6–8], some papers proposed static approaches, using information from a single frame. Therefore these methods can be used to process also vertical scratches in still images. Kokaram  proposed a 2-dimensional autoregressive model for scratch detection and removal, without using information from adjacent frames. Bretschneider et al.  proposed a technique based on wavelet decomposition. Bruni and Vitulano  generalized Kokaram's model for scratch detection on the hypothesis that a scratch is not purely additive on a given image. Tegolo and Isgrò  approach is based on the analysis of the statistics of the image grey levels.
3. Scratches in Still Images
The goal of our work is to find a general approach to the problem, to detect and restore scratches regardless of their specific features. Our general "model" of a scratch is a long thin line which runs along all the image with any orientation and color, but which may also have small variations in width and direction, some little interruptions, but no branches. In most cases, scratches in damaged photos are due to the deterioration of the emulsion or the support of the film. Thus, scratches cannot be considered as "digital" defects, as they come from the digitization of printed documents. Nevertheless, once a photo is digitized, scratches become part of the digital photo and can be digitally processed. The first step is to detect the orientation and the position of the scratch into the image. Then our method labels pixels which are part of the detected line scratch, tracing its contour. To our knowledge, no works in literature deal with the problem of "tracking" scratches, as they typically stop detection when they find the position of the (vertical) line. Last, scratches are restored and lost information recovered. Present work expands over our previous works in quasihorizontal scratch restoration  and multidirectional scratch detection .
4. Multidirectional Detection
The approach presented in  is based on a bandpass filtering, to enhance the horizontal components of the image, and a Hough transformation, to detect candidate line scratches. Therefore, scratches which have slopes with more than 30° cannot be detected. This is not enough to achieve our goals. Our extended solution can be divided into three steps: preprocessing, line detection, and contour drawing.
The preprocessing step aims to enhance image features along a set of chosen directions. First, image is grey-scaled and filtered with a sharpening filter (we subtract from the image its local-mean filtered version), thus eliminating the DC component. Tests showed that this solution makes our approach independent of the color of the scratch, and it helps the next steps in detecting its direction.
so that the center of the subband has , coordinates and it is rotated with the same angle. The subbands must be symmetric with respect to the origin; so the angle must be shifted of 90°. The filter order can control the slope of the subband, so that we chose to concentrate the filtering in a precise zone of the spectrum. This explains the use of this bandpass filter instead of a Gabor one.
4.2. Line Direction Detection
4.3. Contour Drawing
Contour-drawing step requires some user intervention. We propose to the user a set of possible scratches, and he has to select one of the candidates for the next steps in the scratch analysis process. Note that scratches in old photos may be originated by heterogeneous causes, so that they can present interruptions, irregularities in width, slope, and color. Digitization adds a further problem: when digitized, defects of the paper become part of the image. During the acquisition of a printed image, typically an interpolation technique is applied; so the intensity value of the pixels varies according to the surrounding background. Therefore, pixels in the same scratch can have very different intensity values, if the scratch passes across darker and brighter areas.
The contour drawing step can be divided into two substeps: scratch "core" detection and region growing. The first step aims to find the skeleton of the scratch, the second one to find its local thickness.
4.3.1. Core Detection
The aim of the core detection is to find the backbone of the scratch in the most accurate way. For this step we work with lines with slope in . If the line slope is outside this interval, the image is 90° shifted in order to bring the scratch line into this case.
and are fixed weights, set, respectively, to 0.75 e 0.25. The influence of these values on the results is shown in Figure 7. If we increase the "local" weight, we increase the probability to deviate from the principal direction, due to possible local maxima (note the left part of Figure 7(d)). The chosen values give best results, within our dataset.
For each column we select the point, between the 5 candidates, with the maximum similarity value. If this value is lower than a fixed threshold (set by experiments to 0.2), no pixels are included into the core, but our method continues searching for other points along the direction , until the end of the image.
The global factor is used to find all the pixels, which are similar to the reference pixel, along the scratch direction. On the other hand, the local factor makes this method flexible to follow little changes in slope. Moreover, points can be included into the core in spite of possible interruptions along the scratch main direction. It is clear that the choice of a "good" reference point is critical: starting from a point that is along the scratch direction, but is not part of the scratch, makes the core detection process work incorrectly.
4.3.2. Region Growing
In this phase we search for all the pixels which belong to the scratch, orthogonally with respect to the core direction. The approach is similar to that used in the core detection: we include pixels upward and downward instead of rightward and leftward.
and are two fixed weights, set by experiments to 0.85 and 0.15. The influence of these values on the results is shown in Figure 7. If we increase the "local" weight, we increase the probability to include "good" pixels into the scratch to be restored (Figure 7(e))
In  we proposed our restoration algorithm to restore quasihorizontal scratches. In this paper we present the extended version for scratches with any orientation.
5.1. Direction Estimation
For each pixel in the scratch mask, we select two windows ( and ), nearby the scratch, one per side. Window size depends on the average gradient of the pixels in the area, as described as follow. A block-matching approach is used to test all the possible candidate vectors that link a block in one side with a block in the other side, starting from the pixel position. Only blocks that are symmetrical with respect to the pixel position are compared, in order to avoid annoying artifacts (i.e., barrel distortion) into the reconstructed area.
5.2. Pixel Filling
and are the components of the estimated direction vector. If up and down are similar, the new pixel value will be one of them. If they are very different, we choose the vertical interpolated value, which is probably in the middle of the other two. This solution introduces no artifacts but only few blurring.
The restored area is then rerotated and replaced into the original damaged area. Finally, a median filter is applied to the boundary of the scratch mask, to remove some residual artifacts. Experiments showed that the quality of the results is independent of the rotation angle.
6. Experimental Results
We tested our method onto two different sets of images: 20 images coming from a database of aerial photos, which present scratches more or less horizontally oriented, 20 crops of old photos which have heterogeneous physical causes, and present scratches with any orientation and width. Some visual results are shown in Figure 12.
CD is the percentage of correctly detected scratches, with respect to the number of the real ones; percentage of not detected (false negatives) is the complementary value.
FP is the percentage of false positives, that is, the number of detected scratches that are not real, with respect to all the candidate scratches.
is the recall, the ratio between the number of pixels in the intersection of the detected and the real scratch and the number of pixels in the real scratch. When it tends to 1, the detected scratch covers the whole real scratch, but it gives no information about pixels outside ; if it tends to 0, detected and real scratch have smaller intersection.
is the precision, that is, the ratio of the number of pixels in the intersection of the detected scratch and the reference scratch , and the number of pixels in . When tends to 0, the whole detected scratches has no intersection with the real one. If it tends to 1, fewer pixels of are labeled outside AR. Nevertheless this parameter will not assure that the real scratch has been covered.
What can be considered the "real" scratch is a critical issue. For each image in our dataset we manually selected the pixels which belong to a scratch and considered the obtained masks as the "real" contour of the scratch. A manual segmentation process is highly subjective, but it is the only one solution to give a numerical evaluation to our contour drawing method. As expected, we had best results for aerial photos since all the scratches in that dataset have more regular features: horizontal direction, white color, and constant width. We noted that many false positives are real lines, which are indistinguishable, for an automatic inspection method, from line scratches.
Average execution time of the steps of our method.
Average execution time (sec)
6.1. Comparison with Other Works
The choice of a reference method to which we compare our results was a very difficult task. To our knowledge, there are no other approaches which can be applied to our dataset, as none of them deal with the "general" scratch, as we intended. A similar problem is that discussed by Giakoumis et al.  which presents a method to process cracks in damaged paintings. Cracks typically result from mechanical/chemical causes as nonuniform contraction in the canvas or wood-panel support of the painting, the evaporation of volatile paint, or external causes, for example, vibrations and impacts. With respect to cracks, which usually have low luminance and thus can be considered as local intensity minima with rather elongated structural characteristics, scratches may have different aspects (Figure 12 shows the differences between these two types of defects in paintings).
Nevertheless, to further evaluate our experiments, we decided to test our detection method with 5 old movie frames and to compare results with those obtained using the approach proposed in . The reference method is chosen as it is one of the most cited between those which use only spatial information.
7. Conclusions and Future Works
The problem of restoring scratches in still images is much harder to deal with than in old movies, which are typically affected by long vertical thin lines. Scratches in still images can have different, and variable, orientations, thickness, and colors. Furthermore, in still images, no temporal information can be used for the detection. Our method introduces several new contributions with respect to the state of the art. First, our method can detect scratch lines regardless of their orientation. Second, we label pixels that belong to scratch, drawing the defect contour, detecting interruptions, changes in width, and little changes in slope. Moreover tests showed that our method is independent of the scratch color.
The main drawback in the detection step is that it requires user intervention, to select true scratches between a set of candidates, and the starting point. Actually we are working to make our method as more automatic as possible, to select the best starting pixel to draw the contour.
As regards restoration, we proposed a new method, based on block matching. This method ensures the correct reconstruction of lines across the scratch, introducing much less blurring compared to a simple interpolation process. Moreover, our restoration method is very fast, compared with classical inpainting methods, which are typically time consuming. In fact we match only symmetrical blocks, drastically reducing the number of matches to compute. Nevertheless, this method works well only if there a kind of simmetry between the two sides of the scratch, that is, in most cases, a realistic hypothesis, if the scratch thickness is not very large. In case of highly-random textured area, no matching can be found between blocks and vertical interpolation is preferred, and some blurring is introduced. Texture synthesis methods, in these cases, should perform better.
The authors wish to thank the Alinari Photo Archives in Florence and SAS S.R.L. in Palermo for having permitted the use of their photos in this research. They also want to acknowledge all the students who collaborate with them for their implementation work.
- Abas FS, Martinez K: Craquelure analysis for content-based retrieval. Proceedings of the 14th International Conference on Digital Signal Processing, July 2002, Santorini, GreeceGoogle Scholar
- Giakoumis I, Nikolaidis N, Pitas I: Digital image processing techniques for the detection and removal of cracks in digitized paintings. IEEE Transactions on Image Processing 2006,15(1):178-188.View ArticleGoogle Scholar
- Stanco F, Tenze L, Ramponi G: Virtual restoration of vintage photographic prints affected by foxing and water blotches. Journal of Electronic Imaging 2005.,14(4):Google Scholar
- Ramponi G, Bruni V: Virtual restoration of faded photographic prints. Proceedings of 14th European Signal Processing Conference, September 2006, Florence, ItalyGoogle Scholar
- Ardizzone E, De Polo A, Dindo H, Mazzola G, Nanni C: A dual taxonomy for defects in digitized historical photos. Proceedings of the 10th International Conference on Document Analysis and Recognition (ICDAR '09), July 2009 1166-1170.Google Scholar
- Joyeux L, Buisson O, Besserer B, Boukir S: Detection and removal of line scratches in motion picture films. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '99), June 1999 548-553.Google Scholar
- Güllü MK, Urhan O, Ertürk S: Scratch detection via temporal coherency analysis and removal using edge priority based interpolation. Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS '06), May 2006 4591-4594.Google Scholar
- Tenze L, Ramponi G: Line scratch removal in vintage film based on an additive/multiplicative model. Proceedings of IEEE/EURASIP Workshop on Nonlinear Signal and Image Processing (NSIP '03), June 2003, Grado, ItalyGoogle Scholar
- Kokaram A: Detection and removal of line scratches in degraded motion picture sequences. Proceedings of the 8th European Signal Processing Conference, September 1996 1: 5-8.Google Scholar
- Bretschneider T, Kao O, Bones P: Removal of vertical scratches in digitised historical film sequences using wavelet decomposition. Proceedings of the Image and Vision Computing New Zealand (IVCNZ '00), 2000 38-43.Google Scholar
- Bruni V, Vitulano D: A generalized model for scratch detection. IEEE Transactions on Image Processing 2004,13(1):44-50. 10.1109/TIP.2003.817231View ArticleGoogle Scholar
- Tegolo D, Isgrò F: Scratch detection and removal from static images using simple statistics and genetic algorithms. Proceedings of the IEEE International Conference on Image Processing (ICIP '01), October 2001 265-268.Google Scholar
- Ardizzone E, Dindo H, Gambino O, Mazzola G: Scratches removal in digitised aerial photos concerning sicilian territory. Proceedings of the 14th International Conference on Systems Signals and Image Processing (IWSSIP '07), June 2007 393-396.Google Scholar
- Ardizzone E, Dindo H, Mazzola G, Scriminaci M, Vitali M: Multi-directional detection of scratches in digitized images. Proceedings of the 17th European Signal Processing Conference (EUSIPCO '09), August 2009, Glasgow, Scotland 248-252.Google Scholar
- Kass M, Witkin A: Analyzing oriented patterns. Computer Vision, Graphics and Image Processing 1987,37(3):362-385. 10.1016/0734-189X(87)90043-0View ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.