Research of segmentation method on color image of Lingwu long jujubes based on the maximum entropy

Abstract

This paper researches on methods of the color image segmentation method of Lingwu long jujubes based on the maximum entropy to achieve the accuracy of image segmentation and improve accuracy of machine recognition. According to law between the color of Lingwu long jujubes and characteristic of environment, starting from the hue information, this paper is first to explore the difference between the hue of Lingwu long jujubes and the environment which it lives and then use maximum entropy to segment image. It finds optimal threshold by mathematical criterion judging the accuracy of image segmentation. The method of pre-processing of image is mean filter firstly. Then, it extracts hue information of true color image and uses maximum entropy for image segmentation, judging accuracy of image segmentation by segmentation area whether it is in accordance with the 3σ principle. Mathematical morphology is used for smoothing image and eliminating small holes. Finally, segmented image will be obtained through labeling the image by using methods of labeled image and using characteristic parameters for extracting feature. By comparing the segmentation effect with artificial method of the 30 Lingwu long jujubes images, it proves that the color image segmentation method of Lingwu long jujubes based on the maximum entropy has good effect to extract the object region. The accuracy of segmentation rate is up to 89.60%. The time that the algorithm run is 1.3132 s.

1 Introduction

Lingwu long jujubes are important economical fruits in Ningxia. To improve packing efficiency and reduce labor intensity, there is an increasing demand for automatic picking techniques. The identification of fruits plays a key role in automatic packing. Not only is image segmentation basic of image processing which effects directly the performance of image recognition but also it is a difficulty for digital image processing. Sweden, Canada, and Finland research image segmentation algorithms based on machine vision [1,2,3,4,5,6,7]. No algorithm, however, is aimed at Lingwu long jujubes. Existing approaches on image segmentation and feature extraction for Lingwu long jujubes have many disadvantages. Although image recognition of fruits based on machine vision starts relatively late, it develops rapidly. Professor Jiang-ming Kan from Beijing forestry university researches computer vision-based method of automatic measurement of trunk and branch diameters of standing trees and three-dimensional reconstruction [8, 9]. Doctor Yu-tan Wang studies on Lingwu long jujubes’ localization and maturity recognition [10]. Algorithm should be effectiveness or high efficiency [11, 12]. This paper attempts to use improved image segmentation algorithm based on maximum entropy method in image recognition of Lingwu long jujubes, which provides more selection and comparison for image segmentation algorithm of Lingwu long jujubes in the process of automatic picking.

According to actual picking environment of Lingwu long jujubes, this paper finds the rules between hue of Lingwu long jujubes and the surrounding environment and extracts hue information of true color image. Then, it puts forward that the way of combining maximum entropy method and mathematical criterion selects adaptively optimal threshold for image segmentation by analyzing hue histogram of a set of images. Finally, aiming at disadvantages of the image after segmentation, mathematical morphology, labeling, and future selection are used for post processing to extract accurate target feature.

2 Materials and methods

2.1 Image acquisition of Lingwu long jujubes

Hardware equipment of image acquisition includes Levono mobile phone and HP PC computer. For the convenience of experiment, this paper sets the revolution of images to 340 * 300. The type of mobile phone is Lenovo P700 and eight million pixels. The parameters of PC are as follows: HP Pavilion g series, Inter(R) Core(TM) i3-2310M CPU @ 2.10 GHz 2.10 GHz, RAM 2.00, and Windows 7 ultimate ×32. The image is collected under environment of sunny and stored in JPG.

2.2 Pre-processing of image

The image will be polluted by various noises in the process of acquisition and transmission, which affects seriously the effect of image segmentation in post processing because noises are amplified probably. This paper uses 3 × 3 mean filtering [13] for image smoothing to eliminate some noises (Fig. 1).

Fig. 1

Original image. a Image with one jujube. b Image with three jujubes

The distribution of Z neighborhood space in the position of (i, j) is following Table 1.

Table 1 Pixel neighborhood distribution

$$ g\left( i, j\right)=\frac{1}{9}{\displaystyle \sum_{x\in Z}{\displaystyle \sum_{y\in Z} f\left( i+ x, j+ y\right)}} $$

(1)

The pixel value in the position of (i, j) is replaced with g(i, j).

As shown in Fig. 2, two images containing noises are filtered by 3 × 3 mean filtering.

Fig. 2

Image with pretreatment. a The result of image with one jujube after pretreatment. b The result of image with three jujubes after pretreatment

The method, processing channel R, G and B by the same template, has a little effect on each channel, avoiding effectively the image distortion caused by neglecting the link among channels.

2.3 Extraction of hue

It can be learned from the image of mature Lingwu long jujubes that there is the difference between Lingwu long jujubes and its surrounding environment in hue. Figure 3 shows the pixel-value cross sections along line segments which run through part of foreground image and background image. It shows that the value of R component is higher than G component and B component about 20 dB in the position of foreground image and part of background image.

Fig. 3

Line graph of color component. a Color component of image with one jujube. b Color component of image with three jujubes

As shown in Fig. 4, the results of image segmentation are based on the above statistical law.

Fig. 4

Image segmentation based on color component. a Image with one jujube segmentation. b Image with three jujube segmentation

It can be learned that the segmented images based on statistical law are a set of defects. For example, parts of Lingwu long jujubes are neglect. Target image contains a lot of lying in background image, with serous adhesion between target image and background image. However, based on the statistical facts, the R component of Lingwu long jujubes has the dominant position. That is to say that the hue of the Lingwu long jujubes is different from others with tendency of red. Therefore, the hue information extracted that can be obtained by converting the color space is used for post processing. As shown in the formula [14], this paper extracts hue by transforming RGB color space into HSV color space.

$$ \left\{\begin{array}{l} V= \max \left( R, G, B\right)\hfill \\ {}\begin{array}{cc}\hfill S=\frac{\mathrm{mm}}{V},\hfill & \hfill \mathrm{mm}= \max \left( R, G, B\right)- \min \left( R, G, B\right)\hfill \end{array}\hfill \\ {}\begin{array}{ccc}\hfill R\hbox{'}=\frac{V- R}{\mathrm{mm}},\hfill & \hfill G\hbox{'}=\frac{V- G}{\mathrm{mm}},\hfill & \hfill B\hbox{'}=\frac{V- B}{\mathrm{mm}}\hfill \end{array}\hfill \\ {} h=\left\{\begin{array}{l}\begin{array}{cc}\hfill 5+ B\hbox{'},\hfill & \hfill R= \max \left( R, G, B\right)\& G= \min \left( R, G, B\right)\hfill \end{array}\hfill \\ {}\begin{array}{cc}\hfill 1- G\hbox{'},\hfill & \hfill R= \max \left( R, G, B\right)\& G\ne \min \left( R, G, B\right)\hfill \end{array}\hfill \\ {}\begin{array}{cc}\hfill 1+ R\hbox{'},\hfill & \hfill G= \max \left( R, G, B\right)\& B= \min \left( R, G, B\right)\hfill \end{array}\hfill \\ {}\begin{array}{cc}\hfill 3- B\hbox{'},\hfill & \hfill G= \max \left( R, G, B\right)\& B\ne \min \left( R, G, B\right)\hfill \end{array}\hfill \\ {}\begin{array}{cc}\hfill 3+ R\hbox{'},\hfill & \hfill B= \max \left( R, G, B\right)\& G= \min \left( R, G, B\right)\hfill \end{array}\hfill \\ {}\begin{array}{cc}\hfill 5- R\hbox{'},\hfill & \hfill \hfill \end{array}\hfill \end{array}\right.\hfill \\ {} H= h\times {60}^{\circ}\hfill \end{array}\right. $$

(2)

As shown in Fig. 5, hue component is extracted by library function, MATLAB’s own function.

Fig. 5

Image of hue component. a Image with one jujube of hue component. b Image with three jujubes of hue component

2.4 Deflection of hue

The value of hue is described in terms of angle in HSV color space, and it ranges from 0° to 360°, which is characterized by the end to end. That results in the values in the area of 0°+ and 0°− have difference although the color is similar near the area of 0°. That is the reason why images of hue component appear the phenomenon which the part of 0°+ is partial black and 0°− is partial white. This paper will make the hue rotate certain angle solves the problem. As shown in Fig. 6, it is the result of hue deflection.

Fig. 6

Image after hue deflection. a Image with one jujube after hue component. b Image with three jujubes after hue component

2.5 Maximum entropy method and mathematical criterion

2.5.1 Maximum entropy method [15]

Assuming that the number of gray levels of an image is L, the gray scale of the image is {0, 1, 2, ⋅ ⋅ ⋅ ⋅ ⋅⋅, L − 1}. The number of pixels with gray scale k is n _k . The total number of pixels of the image is

$$ N={\displaystyle \sum_{k=0}^{L-1}{n}_k} $$

(3)

The probability of the occurrence of gray is

$$ {p}_k={n}_k/ N $$

(4)

In the above formula, p _k  ≥ 0 and ${\displaystyle \sum_{k=0}^{L-1}{p}_k}=1$.

Threshold T is chosen to classify images into two categories: D ₁ = {0, 1, ⋅ ⋅ ⋅ ⋅ ⋅⋅, T} and D ₂ = {T + 1, ⋅ ⋅ ⋅ ⋅ ⋅⋅, L − 1}, where T ∈ {0, 1, ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ L − 1}. The entropy of class D ₁ and class D ₂ are

$$ {E}_1={\displaystyle \sum_{k=0}^T\frac{p_k}{P{\mathrm{th}}_1}} \log \frac{p_k}{P{\mathrm{th}}_1} $$

(5)

$$ {E}_2={\displaystyle \sum_{k= T+1}^{L-1}\frac{p_k}{P{\mathrm{th}}_2}} \log \frac{p_k}{P{\mathrm{th}}_2} $$

(6)

where $P{\mathrm{th}}_1={\displaystyle \sum_{k=0}^T{p}_k}$, Pth₂ = 1 − Pth₁.

The entropy of the image is

$$ E(T)={E}_1+{E}_2 $$

(7)

The entropy is used as the criterion to judge the effect of segmentation images in different threshold (Fig. 7). What the entropy is larger means the more information is contained in the segmented image. In other words, it has the better segmentation result. Therefore, the optimal threshold

Fig. 7

The three-dimensional histogram of hue. a Three-dimensional histogram of image with one jujube. b Three-dimensional histogram of image with three jujubes

$$ {T}^{*}=\underset{\left(0\le T\le L\right)}{ \max } E(T) $$

(8)

2.5.2 Mathematical criterion

3σ criterion is also called the Lagrandian criterion. When the experimental data are normal or nearly normal distribution, it can be determined by a certain probability interval that is desired data.

In the normal distribution, σ represents standard deviation and μ represents the value of mean. x = μ is the axis of symmetry.

The mathematical (3σ) criterion [16]:

• The probability of the numerical distribution is 0.6826 in the area (μ − σ, μ + σ).

• The probability of the numerical distribution is 0.9544 in the area (μ − 2σ, μ + 2σ).

• The probability of the numerical distribution is 0.9974 in the area (μ − 3σ, μ + 3σ).

It can be considered that the probability is almost entirely concentrated in the interval (μ − 3σ, μ + 3σ) beyond the possibility of this range accounted for less than 0.3%.

For the images of Lingwu long jujubes, the histogram of its hue is obtained, as shown in Fig. 8.

Fig. 8

Hue histogram. a Hue histogram of image with one jujube. b Hue histogram of image with three jujubes

As can be seen from Fig. 8, data distribution of each peak approximate normal distribution in the histogram. The mean and standard deviation of class D ₁ respectively is

$$ {M}_1={\displaystyle \sum_{k=0}^T\raisebox{1ex}{$ k\bullet {p}_k$}\!\left/ \!\raisebox{-1ex}{$ P{\mathrm{th}}_1$}\right.} $$

(9)

$$ S{d}_1=\left({\displaystyle \sum_{k=0}^T{\left( k-{M}_1\right)}^2\bullet {p}_k}/ P{\mathrm{th}}_1\right) $$

(10)

The mean and standard deviation of class D ₂ respectively is

$$ {M}_2={\displaystyle \sum_{k= T+1}^{L-1}\raisebox{1ex}{$ k\bullet {p}_k$}\!\left/ \!\raisebox{-1ex}{$ P{\mathrm{th}}_2$}\right.} $$

(11)

$$ S{d}_2={\left({\displaystyle \sum_{k= T+1}^{L-1}{\left( k-{M}_2\right)}^2\bullet {p}_k}/ P{\mathrm{th}}_2\right)}^2 $$

(12)

Distinguishing the value function of different objects

$$ \left|{M}_1-{M}_2\right|\ge \alpha \left( S{d}_1+ S{d}_2\right) $$

(13)

It can be considered that two objects can be distinguished when gray distribution meets formula (13). The value of α can be adjusted properly according to different condition. The value of α is less than 3.

2.5.3 Adaptive adjustment of threshold

It can be seen from the adjusted histogram that the gray value of the hue component is in the smallest part compared with other parts, which is convenient for the threshold adaptive adjustment to segment the target correctly.

Image segmentation is the process of distinguishing target objects from background objects. It is possible to use multiple maximum entropy method to segment the target objects correctly when more than two different regions need to be segmented in the image. Each segmentation needs to judge the position of target regions relative to the threshold. It may be found how to select the threshold that the hue of Lingwu long jujubes is located in the smallest part in hue histogram.

According to mathematical (3σ) criterion, it is considered that the two targets can be distinguished when its gray distribution satisfies formula (13). There are multiple targets in the image in the process of actual segmentation. Therefore, the image often needs to be segmented several times for segmenting the target objects. But there is a case where the target objects are not divided and the formula (13) is satisfied. In other word, whether the regions segmented are target objects or not, the formula (13) is satisfied when two regions can be distinguished.

This paper transforms the value function from formula (13) to formula (14) according to the characteristics of the hue of Lingwu long jujubes.

$$ J=\left|{M}_1-{M}_2\right|-\alpha \left( S{d}_1+ S{d}_2\right) $$

(14)

Two objects can be distinguished in the condition J ≥ 0.

Assuming J(q) is the value of value function when image is segmented q times, when it meets the relation,

J(q) > 0 and J(q + 1) < 0 or

J(q − 1) < 0 and J(q) > 0

T corresponding to the J(q) is taken as the division threshold.

$$ I(t)=\left\{\begin{array}{c}\hfill \begin{array}{cc}\hfill 1\hfill & \hfill I\_\mathrm{gray}\ge T\hfill \end{array}\hfill \\ {}\hfill \begin{array}{cc}\hfill 0\hfill & \hfill I\_\mathrm{gray}< T\hfill \end{array}\hfill \end{array}\right. $$

(15)

The results of the segmented image are shown in Fig. 9.

Fig. 9

Segmentation results of Lingwu long jujubes. a Segmentation result of image with one jujube. b Segmentation result of image with three jujubes

Generally, number 1 represents target objects and number 0 represents background objects. So there need be transformation of image complement, as shown in Fig. 10.

Fig. 10

Image with complementary. a Image with one jujube after complementary. b Image with three jujubes after complementary

2.6 Post processing of image segmented

For post processing of image, there are several methods including mathematical morphology, labeling, and feature selection.

2.6.1 Mathematical morphology

Mathematical morphology contains four basic operations: dilation, erosion, open, and close.

The dilation is used to smooth the boundary and fill the small void, the formula

$$ X\oplus B=\left\{ a\Big|{B}_a\cap X\ne \varPhi \right\}=\left\{ a\Big|{B}_a\uparrow X\right\} $$

Erosion is used to eliminate small and nonsensical points, the formula

$$ X\varTheta B=\left\{ a\Big|{B}_a\subset X\right\} $$

Open is used to eliminate small and meaningless points and separate the target object of adhesion, the formula

$$ X\circ B=\left( X\varTheta B\right)\oplus B $$

Close is used to fill small holes, connecting adjacent objects, the formula

$$ X\cdot B=\left( X\oplus B\right)\varTheta B $$

As a result of the segmented image, a part of the background objects is also divided into target objects. At the same time, there are bad phenomena including small holes in the image and adhesion between background objects and target objects in some location. Based on the above situation, this paper uses open algorithm to eliminate the small and meaningless points, with the separation of the target object of adhesion; the effect is shown in Fig. 11.

Fig. 11

Results using mathematical morphology. a Segmentation result of image with one jujube by using mathematical morphology. b Segmentation result of image with three jujubes by using mathematical morphology

2.6.2 Labeling and feature selection

Labeling is a method of processing an area in which the same areas are marked with the same marks and the different areas are marked with different marks.

The feature parameters include the area, perimeter, compactness, moment, and eccentricity. Different objects can be differentiated and extracted by using those feature parameters.

In this paper, the area is selected as the characteristic parameter extracting target. Firstly, the different object in the image is marked by the method of labeling, and then, the area of each object is calculated to extract the target objects through threshold set.

3 Algorithm steps

In order to improve the accurate segmentation rate of Lingwu long jujubes, this paper comprehensively uses mathematical criterion, mathematical statistical analysis, maximum entropy method, and feature selection to segment the final target objects successfully (Fig. 12). The algorithm flow chart is shown in Fig. 13.

Fig. 12

Segmentation results using algorithm of this paper. a Final segmentation result of image with one jujube by using algorithm of this paper. b Final segmentation result of image with three jujubes by using algorithm of this paper

Fig. 13

Flow chart of algorithm of this paper

Key steps:

• Step 1: input original image.

• Step 2: pre-processing of original image.

• Step 3: extract hue information.

• Step 4: maximum entropy method is used to obtain threshold T in the range from 0 to G, then calculate the mean M1,M2 and standard deviation Sd1, Sd2.

• Step 5: calculate J(p) by J(p) = |M ₁ − M ₂| − α(Sd ₁ + Sd ₂).

• Step 6: judging whether (J(p) > 0 and J(p + 1) < 0) or (J(p-1) < 0 and J(p) > 0) when cycle two or more time. If J(p) meets the criteria, go to step 7. If not, let G = T and go to step 4.

• Step 7: image is segmented though the threshold which J(p) obtained is corresponding to T.

• Step 8: post processing by using mathematical morphology.

• Step 9: output image segmented.

4 The result of experiment and analysis

In this paper, 30 images of Lingwu long jujubes are segmented. The reliability of the algorithm is tested by the wrong segmentation rate. The statistical results are shown in Table 2.

Table 2 Date of segmentation of Lingwu long jujubes

The number of pixels in the artificial segmentation is calculated by Photoshop which extracts the target objects of Lingwu long jujubes. The number of pixels extracted wrongly is the difference between region in the artificial segmentation and region extracted correctly by the algorithm (Fig. 14). Wrong rate is a ratio that takes into account the numbers of pixels segmented wrongly and the numbers of pixels in the artificial segmentation. It can be learned from the table that the wrong rate is 10.40% and the correct rate is 89.60%. For different images that have similar background, the results are not same. The highest error rate can reach 28.19%, while the smallest error rate is only 3.06%. On the experiment of 8th, 13th, 15th, and 21st, the accuracy of the segmentation is different although those images have similar ratio between the area of Lingwu long jujubes and the area of whole image. The rate of wrong segmentation is 28.19% on the experiment of 8th, while the others are less than 10%. For no. 8, no. 17, and no. 27, the wrong rate is more than 20%.

Fig. 14

Results which were not segmented well

The possible reasons that appear in the above phenomenon are as follows:

• 1. The image segmentation is performed by the hue information, and the more cyan region of Lingwu long jujubes is not identified.

• 2. There is the same trend of hue information among adjacent regions in target objects and background objects.

The average running time of 30 similar images is 1.3132 s. Compared to image in the artificial segmentation, the effect of segmentation by algorithm of this paper is following Fig. 15c, d.

Fig. 15

Comparison of artificial segmentation and new method of the paper. a Segmentation of image with one jujube by algorithm of this paper. b Segmentation of image with three jujubes by algorithm of this paper. c Artificial segmentation of image with one jujube. d Artificial segmentation of image with three jujubes

It can be learned from 14 (a), 14 (b), 14 (c) and 14 (d), both images segmented of the new algorithm and images of the artificial segmentation have the same effect of segmentation, with accurate regions of extraction except the small error in the local area.

5 Discussion and conclusions

In this paper, the feature of Lingwu long jujubes can be extracted well, but the dependence of feature extraction is large:

• 1. The feature extraction depends on the red range of the image and not consider the blue part of the date.

• 2. The angle of rotation is uncertainly. If hue angle of rotation is too large, the local hue value of the date changes greatly, which is not conducive to the segmentation.

• 3. Uncertainly of parameters in mathematical criterion. It is possible that the same parameter may have a large difference in the two types of divisions with respect to the slight difference in hue.

Based on the characteristics of the growth environment of Lingwu long jujubes, this paper is first to make statistical analysis of information to find out the difference between Lingwu long jujubes and other parts in the image and then extract the color information by transforming color space from RGB and HSV. Combining the distribution characteristics of red, the hue is rotated at certain angle. By analyzing the histogram of hue information, there are many valleys and peaks and each mountain peak is approximately normal distribution. There are many valleys and peaks, and each mountain peak is approximately normal distribution by analyzing the histogram of hue information. Therefore, the maximum entropy method and mathematical criterion are adopted to adaptively select the appropriate threshold. Finally, mathematical morphology, labeling, and feature selection are used to post processing to extract accurate target objects. A set of mathematical methods such as statistical analysis, maximum entropy, mathematical criteria, and mathematical morphology are used in this paper, which can well control the adaptive threshold selection and find the optimal threshold. The correct rate of segmentation is 89.60% through the test of 30 similar images of Lingwu long jujubes. In the literature [17, 18], the segmentation of the Lingwu long jujube image is based on the L * a * b color space and the color difference fusion. The correct segmentation rate of Lingwu long jujube is over 92.6% and the running time is 2.1889s. In this paper, the average running time, which is higher than 0.8757 s, is 1.3132 s. Although it is a little lacking in the correctness of segmentation, it greatly improves the operation speed. This method can meet the requirement of image real-time processing and can provide the theoretical basis for intelligent harvesting of Lingwu jong jujubes based on the requirement of segmentation accuracy.