Multifractal characterisation and classification of bread crumb digital images

Abstract

Adequate models of the bread crumb structure can be critical for understanding flow and transport processes in bread manufacturing, creating synthetic bread crumb images for photo-realistic rendering, evaluating similarities, and establishing quality features of different bread crumb types. In this article, multifractal analysis, employing the multifractal spectrum (MFS), has been applied to study the structure of the bread crumb in four varieties of bread (baguette, sliced, bran, and sandwich). The computed spectrum can be used to discriminate among bread crumbs from different types. Also, high correlations were found between some of these parameters and the porosity, coarseness, and heterogeneity of the samples. These results demonstrate that the MFS is an appropriate tool for characterising the internal structure of the bread crumb, and thus, it may be used to establish important quality properties it should have. The MFS has shown to provide local and global image features that are both robust and low-dimensional, leading to feature vectors that capture essential information for classification tasks. Results show that the MFS-based classification is able to distinguish different bread crumbs with very high accuracy. Multifractal modelling of the underlying structure can be an appropriate method for parameterising and simulating the appearance of different bread crumbs.

1 Introduction

The goals of this research are (1) to evaluate if the MFS [1] can be applied to characterise and discriminate the bread crumb structure for different bread types from digital images and (2) to investigate the effectiveness of the method in the classification of these structures.

One of the most important factors to evaluate the quality of a bread loaf is related to its crumb structure. Close examination of different slices reveals considerable variation in the cell (air bubble) size even within a single sample of the same bread type.

Fractal and multifractal analysis of images has proved to be able to capture useful properties of the underlying material being represented. These features have been successfully applied in different areas, such as medicine [2,3] and texture classification [4]. In food research, fractal and multifractal analysis has been applied in the study of apple tissues [5], pork sirloins [6], and also in chocolate, potato, and pumpkin surfaces [7]. Through several procedures [8,9], it is possible to obtain different fractal dimensions (FD), each of them capturing a different property of the material (e.g. porosity, rugosity).

Data analysis of the results of the feature extraction process is useful for obtaining key properties of materials. This information could then be used in quality measurements of real samples and in the validation of synthetic representations of them. In other words, these processes are useful to determine if a given image presents the observed features in that material, allowing to associate quality measure parameters to it. In [10], a bread crumb quality test based on Gabor filters was performed, obtaining good quality assessment. Nevertheless, a small database was used (30 images). In [9], several fractal features were obtained for one type of bread, demonstrating that a vector of FDs would be capable of obtaining key features of the crumb texture more accurately than using a single FD.

In this work, we propose the application of the multifractal spectrum (MFS) to describe and classify different bread crumb types. One of the main features of the MFS is its bi-Lipschitz invariance, that is, invariance to perspective transforms (viewpoint changes) and smooth texture surface deformations. It is shown that the MFS is also locally invariant to affine changes in illumination. In other words, MFS analysis is in theory a robust feature extractor, what makes it specially adequate for the purposes of our study.

Food classification has already been applied using fractal and other techniques in [11,12], but these works do not address the intra-class problem, i.e. the classification is made among different foods and not by making different classes out of the same food.

In a previous work [13], we showed that the MFS in combination with other fractal features was able to classify different bread crumb types with high accuracy. The present work aims to simplify the model and strengthen these results by comparing only the MFS with other state-of-the-art features and using different classifiers and also to study the correlations of the features obtained with the procedure and different texture features obtained from the images.

The proposed method is compared to other state-of-the-art features for texture classification. The results of this feature extraction procedure show that the classifier is robust and presents good discrimination properties to distinguish between different bread types and also bread from non-bread images.

This paper is organised as follows. In Section 2, the theory underlying fractal sets is introduced, and the materials and methods employed in this work are presented. In Section 3, the results obtained in the characterisation and classification procedures are shown and discussed. In Section 4, the conclusions are summarised, as well as possible future works.

2 Materials and methods

2.1 Fractals and multifractals

The term fractal was first employed by the mathematician B. Mandelbrot in [14]. Fractal objects have the property of self-similarity (i.e. the geometrical or topological properties are invariant at different scales), and they are characterised by a non-integer dimension. Fractal objects can have one or more FDs. Most of the famous fractal sets (i.e. the Cantor set, the Von Koch curve, and the Sierpinski gasket) can be characterised by a single exponent that relates how their geometrical properties vary under scale changes. On the other hand, there are cases where the fractal object exhibits different exponents under different scales. Those are called multifractals [15] and are characterised by a sequence of FDs or even a function that establishes the local variance of the geometrical properties under scale changes. It is assumed that these structures are composed by different fractals coexisting simultaneously. The self-similarity, then, can be characterised by a multifractal spectrum that establishes the specific fractal behaviour of the set at a given scale. The multifractal approach characterises better the objects than the fractal one, since variations in local regions are captured in a more accurate manner. A particular definition of dimension, the so-called box dimension, is employed in fractal and multifractal analysis.

2.1.1 2.1.1 Box dimension

On the one hand, mathematical objects such as the Koch curve and the Sierpinski triangle have exact self-similarity. Natural phenomena, on the other hand, are better described by statistical self-similarity. In such cases, the box FD is used. Box FD is a simplification of the Hausdorff (originally Minkowski-Bouligand) dimension for non-strictly self-similar objects [8]. Given an image I, it is subdivided in a regular grid of boxes of side ε. If N _ε represents the amount of boxes that contain at least one point of I for that ε, then the box dimension D _b is defined as:

$$ D_{b} \triangleq {\lim}_{\varepsilon \to 0}{\frac{\log\left(N_{\varepsilon}\right)}{\log \left(1/\varepsilon\right)}}. $$

((1))

A practical algorithm for computing D _b in digital binarised images establishes different partitions of the original image in regular grids of side ε (in pixels) and counts for each partition the amount N _ε of boxes that contains at least one pixel of the object of interest. It uses a binarised image and selects different values of ε in it, making a count of the boxes that contains pixels in each case. Since this procedure is not invariant under translations, an average of grid counting is performed under several translations at a given grid size ε. With the previous steps, a set of {1/ε,N _ε } pairs is obtained, with which a linear fit is performed in log-space. Finally, the slope of the resulting straight line is by definition the box dimension of the image. In Figure 1, an image of the bread type bran is shown (left), with an example of binarisation (centre) and its corresponding box dimension computation (right).

Figure 1

Box dimension computation. A bread image (left) with its binarisation (centre) and its computed box dimension (right).

2.2 The theory behind multifractal analysis

In [9], several procedures were applied to analyse the bread crumb structure showing that a vector of FDs could better characterise those structures. Based on that assumption, in this work, a multifractal analysis of the bread crumb is carried out. The idea behind multifractal analysis is to examine, in the limit, the local behaviour of a measure μ at each point of the set under study.

Let E be a structure divided in disjoint substructures E _i of size ε in such a way that:

$$ \bigcup_{i}E_{i} = E. $$

((2))

Each substructure E _i is characterised by a measure μ(E _i ). From the point of view of multifractal analysis, it is useful to define the Hölder exponent, α _i , for each substructure E _i , as a function of ε, i.e.,

$$ \alpha_{i} \triangleq \frac{{\text{ln}}\left(\mu(E_{i})\right)}{{\text{ln}}\left(\varepsilon\right)}, $$

((3))

and to take the limit when ε tends to 0. The limit α represents the value of the Hölder exponent at a point in the structure, that is:

$$ \alpha = {\lim}_{\varepsilon\to0}{\alpha_{i}}. $$

((4))

This exponent characterises the local regularity of the structure at a point. To obtain a global characterisation of the regularity of the structure, it is necessary to obtain the distribution of α in E. Then, the multifractal spectrum related to the value of ε, f _ε (α _i ), is obtained by counting the N _ε boxes characterised by α _i , in the form:

$$ f_{\varepsilon}(\alpha_{i}) = - \frac{{\text{ln}}(N_{\varepsilon}(\alpha_{i}))}{\text{ln}(\varepsilon)}. $$

((5))

When ε tends to 0, the limiting value is the FD of the structure E characterised by α (the Hausdorff dimension of the α distribution). It is also known as the multifractal spectrum f(α) (MFS) [16], i.e.,

$$ f(\alpha) = {\lim}_{\varepsilon\to0}{f_{\varepsilon}(\alpha)}. $$

((6))

2.2.1 2.2.1 Practical procedure for the MFS

There are several techniques described in the literature to obtain the MFS, which lead to different representations of the multifractal information present in the structure. Usually, the method of moments is used [5,6], but it produces a feature vector which is not always suitable for classification tasks. In this work, the procedure presented in [1] is employed, due to its better classification performance.

The technique first computes α(x) for each pixel x of the image. Denote with B(x,ε) the closed disk of radius ε>0 centred at x, then, α(x) is defined as a straight line fit of the values log(μ(B(x,ε))) and log(ε). Then, a discrete sample set {α _i ,i=1,…,M} is taken from the interval [ 0,1], and the point set corresponding to that value of α _i is formed by grouping the pixels with values that are close to that α _i under some threshold. The FD for each point set is computed as the straight line fit of the values log((N _ε (α _i )) and log(ε). The value M determines the vector length, i.e. the number of FDs of the MFS.

As previously stated, the f(α) spectrum (MFS) and the method of moments produce vectors which contains the same information, but in this work, the first is employed, since it also outperforms the method of moments in classification tasks. This process produces a finite vector which is used as the feature vector later in this paper. In the next sections, the vector length (the number of FDs) is chosen based on the classification performance of the computed feature vector.

2.2.2 2.2.2 Multifractal measures

Defining different μ functions accounts for different image features. The first approach is to define μ in the intensity domain (I), i.e.,

$$ \mu\left(B(x,\varepsilon)\right) = \int_{B(x,\varepsilon)}{(G_{\varepsilon} \ast I)} dx, $$

((7))

where ∗ is the 2D convolution operator and G _ε is a Gaussian smoothing kernel with variance ε, i.e. μ is the weighted average intensity value in the disk of radius ε centred at x (B(x,ε)). This is the density of the intensity function, and it describes how the intensity at a point changes over scale.

The definition of μ could serve to specific purposes. For instance, if robustness to illumination changes is needed, one choice is to define μ(B(x,ε)) on the domain of the energy of the gradients. Let f _k ,k=1,2,3,4 be four directional differential operators along (respectively) the vertical, horizontal, diagonal, and anti-diagonal directions. Then, we define a measure function μ(B(x,ε)) for the image I as follows:

$$ \mu(B(x,\varepsilon)) = \left(\int_{B(x,\varepsilon)}\sum\limits_{k}\left(\,f_{k} \ast (G_{\varepsilon} \ast I)\right)^{2} dx\right)^{1/2}. $$

((8))

Another choice is to define μ(B(x,ε)) as the sum of the Laplacians of the image inside B(x,ε), that is:

$$ \mu(B(x,\varepsilon)) = \int_{B(x,\varepsilon)}|\nabla^{2} (G_{\varepsilon} \ast I)| dx. $$

((9))

All these alternative measures modify the computed FD and MFS (except for trivial or monofractal images) and therefore are valuable choices in finding adequate features, as will be shown below.

2.3 Image acquisition

For testing our characterisation and classification procedures, 20 images of 4 different commercial bread types (baguette, sliced, bran, and sandwich), counting 80 images, were obtained in the same day of purchase using an HP PSC 1210 scanner with the following settings: highlight 190, shadows 40, and midtones 1, and they were saved in TIFF format. Images were acquired at a resolution of 380×380 pixels (the maximum possible area for the four bread types) and 350 dpi (1 pixel =0.00527[mm²]). Then, the images were converted to grey scale (8 bits). In addition, 20 other images of each bread type were acquired with a digital camera, using the same spatial resolution. The illumination conditions of these images were different from that of the scanner in order to test for the robustness of the method. In Figure 2, four examples of bread images obtained with the camera are shown. We also employed 100 randomly selected images from the CalTech101 dataset [17] in order to test the method’s performance with non-bread images.

Figure 2

Images from a digital camera.

The void fraction (VF), mean cell area (MCA), and standard deviation of mean cell area (stCA) were computed in order to study the relationship of the MFS with the porosity, coarseness, and heterogeneity of the different bread crumbs. For this purpose, the scanned images (in grey scale) were binarised using the algorithm presented in [18]. This algorithm applies a local thresholding scheme, which showed better results than using a global thresholding scheme. Particularly, the algorithm presented in [19] and used in [9] showed poor results when the illumination conditions vary in the image. In Figure 3, an image of each bread type used in this work (top row) and its resulting binarisation using the proposed algorithm (bottom row) are shown. Small elements of 1 and 2 pixels were eliminated by an opening operation (erosion and dilation) using a 2×2 structuring element. The method showed good results even for different illumination conditions varying in the same image.

Figure 3

Images from a scanner.Baguette, sliced, bran, and sandwich bread types (top row) with their corresponding binarisations (bottom row).

In order to determine the result of the binarisation at a given pixel, the algorithm obtains an average from the grey levels in a window surrounding the pixel and compares it to a threshold determined by the actual grey level of the pixel multiplied by a bias factor, i.e.,

$$ \frac{\sum_{x,y \in W} f(x,y) }{W_{\text{size}}} \geq f(x_{c},y_{c}) \times {\text{bias}}, $$

((10))

where x _c ,y _c are the coordinates of the actual pixel, and W is the window surrounding that pixel. Two parameters must be set in the algorithm: the size of the window (W _size) and the bias. It was found that different values for the bias are needed for better results when different capturing methods are used. The optimal values for the scanner samples were 80 for the window size and 1.15 for the bias. In the case of the digital camera samples, the optimal values found were 80 for the window size and 1 for the bias. These differences seem to be caused by the different illumination conditions present in the images resulting from these different capturing conditions. Further research is required in order to determine automatic values for these parameters.

3 Results and discussion

In this section, we will attempt to show how the MFS behave adequately as a feature descriptor able to distinguish bread from non-bread images. For this purpose, the MFS, using 20 FDs, was computed for each of the 200 images (i.e., 40 images of each bread type and 40 randomly selected non-bread images, getting 5 balanced classes). In the next subsections, we show how the computed data is analysed and used for classification purposes.

3.1 Data analysis

Self-organising maps (SOM) [20] of the feature vectors associated with each bread image were useful to represent them in a lower dimensional view, in order to better understand the meaning of their respective MFS. A SOM maps high-dimensional data into a (typically) two-dimensional representation, using neighbourhood information. Topological information of the original data is preserved.

Unsupervised SOM of the multifractal representation of bread and non-bread images are shown in Figure 4 in a grid of 10×10 cells (the behaviour observed is similar for different grid sizes). In the left image, the 5 classes (e.g. baguette, sliced, bran, sandwich, and non-bread) are shown, while in the right image, the non-bread class has been removed, and then, the SOM was recomputed for the remaining 4 classes, in order to highlight details among the MFS of the different bread types. The multifractal feature SOM appears to show easily separable classes. It seems that a classifier could potentially obtain low classification errors using the multifractal features, since the numbers representing different classes are clearly separated to each other, i.e. there are almost no cells with two or more different numbers in it. A classifier can define regions of space (group of cells) for each class.

Figure 4

Self-organising maps (SOM). SOM of the bread and non-bread images (left) and SOM of the bread types only (right). (1)baguette, (2)sliced, (3)bran, (4)sandwich, (5): non-bread.

In Figure 5, boxplots of the four different bread types are shown with the medians of each FD (in red) joined by a dashed line. Each FD corresponds to a value of α _i . In our experiments, the MFS vector has 20 FDs (for reasons that will be explained in the classification section). From this plot, it could be pointed out that in the first half of the MFS (first 10 FDs, α∈[ 0,0.53)), the dispersion of the FDs is higher than in the second half of the spectrum (last 10 FDs, α∈[ 0.53,1]). The spectrum in the last FDs tends to have a shape that identifies better a particular type of image. Usually, the spectra of the same class and the same capturing method have, in this part of the spectrum, a shape that is useful to characterise the class. Nevertheless, the capturing method and the illumination conditions of the image influence this shape, i.e. these two factors alter the MFS of the image. In other words, there is no unique shape for each class of bread type.

Figure 5

Boxplots of the four different bread types. The FD medians are joined by a dashed line. Top: baguette (left), sliced (right), bottom: bran (left), sandwich (right).

For the sake of completeness, in Figure 6, the mean values and the standard deviations of the MFS for the four different bread types are shown. This image shows, similarly to the SOM maps, that the MFS could potentially characterise and classify the different bread crumb types, since the mean values are different for each class. The standard deviations are also consistent with the results found in the boxplots.

Figure 6

Mean MFS and standard deviations of the FDs for the four different bread types. Left: mean MFS, right: standard deviations.

The Spearman correlation coefficient (ρ) for the four bread types between the fractal dimensions and the void fraction, mean cell area, and standard deviation of mean cell area (in [mm²]) are shown in Figures 7, 8, and 9, respectively, as a function of fractal dimension. In order to better describe the results, only the scanned samples were employed to analyse these correlations. ρ is preferred over Pearson’s R coefficient since it does not require a linear dependence to exhibit strong correlations in the underlying data.

Figure 7

Spearman correlation coefficients for the FDs and the void fraction of the scanned samples.

Figure 8

Spearman correlation coefficients for the FDs and the mean cell area. (In [mm²], scanned samples).

Figure 9

Spearman correlation coefficients for the FDs and the standard deviation of the mean cell area. (In [mm²], scanned samples).

In Figure 7, it becomes clear that the coefficients behave similarly for the first 5 dimensions (α∈[ 0,0.23]) in all the bread types but differently for the FD around 5 and above. It could be concluded that the first 5 dimensions are highly correlated with the void fraction (porosity) of the scanned samples. This means that the first FDs increase when the void fraction increases. Other FDs also have a high (positive or negative) correlation, but it depends on the bread type which dimension is correlated.

From the plots of the correlation coefficients of the MCA and stCA, in Figures 8 and 9, respectively, it could also be pointed out that the MCA has a higher correlation than the stCA with the FDs of the MFS. It means that the coarseness of the bread crumb structure could be better characterised by the features than its heterogeneity, using the MFS. In addition, the last 5 fractal dimensions of the spectra (α∈[ 0.79,1]) are highly (inversely) correlated with the MCA of the scanned samples. This implies that the last FDs increase when the MCA decrease. The same observation could be mentioned for the stCA of the samples, but the correlations are lower. In both cases, the correlation coefficients of the sandwich class are the lower among the bread types.

To summarise, the dimensions of the MFS which corresponds to α∈[ 0,0.23] are useful to measure the porosity of the scanned samples. Also, coarseness and heterogeneity could be measured by the dimensions with α∈[ 0.79,1]. As was suggested in a previous work [9], the bread crumb structure is better characterised by the use of a vector of fractal dimensions, since the three properties could be measured by the MFS, employing different sections of this feature vector.

3.2 Bread classification

In order to test for the discriminative capability of the method, a classification experiment is made. Five classes are defined, viz., baguette, sliced, bran, sandwich, and non-bread, assigning 40 images to each class. A comparison is made between the MFS and state-of-the-art features in the computer vision literature. This classification scheme corresponds to an intra-class problem, which is harder to solve than an ordinary inter-class one.

K-fold cross validation is applied to the entire set (with K=4), employing three different classifiers: support vector machines (SVM), random forests (RF) [21], and nearest neighbours (NN). Results show that the MFS presents good classification performance regardless of the classifier employed. The libsvm implementation [22] was used for the SVM classifier (with RBF kernel). In the case of the RF (100 trees) and the NN (1 neighbour) classifiers, the scikit-learn python library was employed.

In Table 1, the classification performance of the method is tested using different numbers of FDs. When 20 FDs are used, a useful combination of performance and low dimensionality is achieved (it shows the best classification results for the RF and NN classifiers), so this number of FDs is used in the following computations.

Table 1 Bread crumb classification results with different numbers of FDs for the MFS and different classifiers

In Table 2, several combinations of different MFS obtained from the images and their classification performance are shown. The MFS used in the study were computed based on the density of the intensity (MFS in the table), the Laplacian of the intensity (L), and the gradient of the intensity (G) (see section ‘Multifractal measures’). In addition, another test is made, using the CIELab [23] colour space. The key advantage of this colour space is that it tends to reduce the dependency of the resulting image colour on the device used in the capture. The intensity of the images is transformed to the CIELab space, and the MFS of the three separated channels (L, a, and b) are combined together, obtaining a vector of 60 FDs. This combination showed the best classification performance. It means that adding colour information in the a and b channels is useful for better classification of different types of bread crumbs, when different capturing devices are used (in this case, a scanner and a digital camera).

Table 2 Bread crumb classification results using different combinations of the MFS and different classifiers

In Table 3, state-of-the-art features (Haralick [24], local binary pattern (Lbp) [25], and SIFT [26] features) are computed for the images. The best classification performance is obtained using the SIFT features, but a feature vector of length 128 is required for every image, and, in addition, computational space and time is needed to build internal structures (e.g. a codebook).

Table 3 Bread crumb classification results for different state-of-the-art features and different classifiers

For better understanding of the classification results, confusion matrices of the classification procedures could be plotted. As an example, the confusion matrix (from the cross validation) of the best results, i.e. the CIELab method, employing the SVM classifier, can be seen in Table 4. In each column of the matrix, the output of the classifier for the 40 images of each class is tested for correctness. The table shows that only in a few cases, the classifier returns an incorrect result. For instance, among the 40 images of sliced, only 2 are classified as images from other classes, specifically as baguette and sandwich (column with heading sliced). The other classes behave similarly. Differentiation between bread and non-bread images has no errors (i.e. no image of a bread class is classified as non-bread and vice versa). The 97.5% of the database is correctly classified (5 misclassifications out of 200).

Table 4 Confusion matrix for the best results (CIELab method, using the SVM classifier)

The classification performance of the MFS for the bread crumb database is the highest among the algorithms studied. The MFS captures robust and useful information for classification in low-dimensional features. These results also agree with results obtained in [12] for the classification of other food products.

4 Conclusions

The visual appearance of different types of bread crumbs can be successfully characterised by the multifractal dimensions of their digital images. The FDs obtained from the MFS method whose α∈[ 0,0.23] provided a good measure of the bread crumb porosity, meaning that the higher these FDs, the higher the measure. In addition, the FDs whose α∈[ 0.79,1] are useful to measure coarseness and heterogeneity of bread crumb. The MFS contains useful data to characterise the three measures, combining the information in one feature vector.

The use of multifractal features in bread crumb texture classification showed excellent performance. The MFS demonstrated to be accurate enough to perform a classification of different bread types and also to discriminate non-bread from bread images. The classification performance of the MFS for the bread crumb database outperforms other state-of-the-art techniques employed in the computer vision literature. The information present in the MFS of the L, a, and b channels of the CIElab colour space obtained the best classification performance in all the developed tests. This result appears to be a consequence of the different capturing devices used in this work. Also, it was shown that the MFS is sensitive to changes in the illumination conditions during image acquisition.

The results found could also be applied to validate synthetic samples, in the sense that they should have similar features to the bread type they are trying to simulate. The features found with the MFS could be employed to tune bread crumb quality parameters.