- Research
- Open access
- Published:
Imagery non-photorealistic art processing in modern product design relying on an inline block optimization hierarchy tree algorithm
EURASIP Journal on Image and Video Processing volume 2018, Article number: 71 (2018)
Abstract
In view of the high complexity of image arts processing in modern product design, this paper proposes an image non-realistic art processing method in modern product design that relies on an inline block optimization hierarchical tree algorithm. This method is adopted in modern product design. In the introduction of a nested block optimization hierarchical tree algorithm, according to the different image block level classification, the statistical autocorrelation function matrix is estimated, followed by the non-realistic art processing of each image block in the product design. Because the algorithm uses an inline block optimization hierarchical tree algorithm, this avoids the non-linear iterative phenomenon in the process of traditional product design image processing, and greatly shortens the image processing time. The simulation experiment results show that the image processing quality of the proposed algorithm is not affected due to the reduced computational complexity for the image with less complex texture details. The reduction of image processing quality affects the image processing algorithms used in modern product design.
1 Introduction
Non-photorealistic processing is not only specific non-realistic processing, but also stylized processing. The focus of such a stylized approach is not to truly reproduce the scene of the objective world. Instead, it focuses more on the image personalization and artistic expression. With the development of computer technology, the purpose of non-photorealistic processing is to design a certain algorithm to perform processing based on the true color images, and finally obtain images with the non-realistic effects. Unlike the photorealistic effects created by photorealistic processing, non-photorealistic processing refers to the use of computer-generated graphics rendering techniques with artistic style. The purpose of non-photorealistic processing is not about the authenticity of the objective world, but mainly focuses on highlighting the artistic quality of objects by using abstract or certain artistic processing styles or highlighting the local details of objects to show the effect that is otherwise impossible to be expressed in photorealistic processing. In recent years, non-realistic art processing (NRAP) has become a rapidly developing field of research and has received extensive attention [1]. Many researchers have done a lot of work in this area. Decarlo et al. completed the processing of non-photorealistic styles with the assistance of hardware devices. However, the use of hardware has limited the flexibility of the application processing. Domestically, Tang Yun et al. put forward a non-photorealistic image simulation method based on scan lines. The core is the contour extraction and color homogenization processing, but experiments have shown that the non-photorealistic processing using this method generates color region transitions and false scanning stripes. Computer graphics technology mainly focuses on simulating the objective world and reproducing what people see accurately. Based on various means, simulation can be used to obtain the image effect that is closest to the real world and has a high degree of truth [2]. Realistic rendering technology has become more and more mature with wide application. With the continuous advancement of computer software and hardware technology, we are faced with images and display with higher and higher resolution. The computer-simulated image becomes increasingly clear and close to the real world [3, 4]. However, when applying NRAP to image processing in modern product design, two major problems are faced: the computational complexity of image processing is too large and the storage of measurement matrices is huge [5]. The computational complexity of problematic image processing is too large. Image processing performance in exchange for computational complexity is reduced, but it is not ideal for image processing in modern product design [6]. As the measurement matrix storage capacity is huge, a structural random matrix is proposed to reduce the storage capacity of the measurement matrix. The whole image can be measured quickly. But the structure random matrix adopts the popular orthogonal transformation matrix (Fourier transform, discrete cosine transform (DCT), Hadamard transform). According to NRAP non-correlation measurement theory, the matrix does not have good generality [7]. There is poor correlation between and independent Gaussian measurement matrix and any fixed matrix, so it can be used to design a common image coding strategy in modern product design. More recently, it was proposed that an inline block measurement and image processing phase combination, while using an independent and identically distributed Gaussian measurement matrix, avoids the huge amount of storage defects, making this method particularly suitable for real-time images [8]. In the framework of the combination of embedded block measurement and image processing, the use of directionality transformation to increase the image sparsity further improves the image processing performance in modern product design. The multi-resolution analysis was introduced to the NRAP measurement process to improve the performance of image processing in modern product design at the expense of increased computational complexity. In the above method, block-based acquisition uses the same measurement matrix, thus ignoring the fact that each block has different characteristics (edges, textures, etc.) and eventually influencing image processing quality. According to the method of adaptively changing the measurement rate of block features, the above defects are effectively overcome and good results are obtained [9]. In addition to a large number of applications for 2D images, NRAP has low memory and fast image processing speed. Its advantages make it also widely used in 3D images: to make effective use of image processing algorithms in modern product design to recover image residuals, to obtain good image processing quality at a low cost, and to use embedded block-based dictionaries. Generate strategy, use the inter-frame correlation to construct a dictionary for each block, enhance block sparsity, and enhance the image processing performance in modern product design; The image coding motion estimation and motion compensation image processing algorithms, designed to introduce modern products, also received good results. In short, it can be seen in the actual image and image applications, that BNRAP has a great advantage [10].
Although the above image processing algorithms in various modern product designs have obtained good image restoration quality, the time required for image processing is still too long, which is still not suitable for real-time applications. The fundamental reason for the slow image processing is that these algorithms are use nonlinear optimization design, the process contains a large number of iterations, and too many iterations affect the image processing time. For this problem, this paper reports on application of the embedded block optimization hierarchical tree algorithm to the image processing of modern product design. This can greatly shorten the image processing time, while still retaining the high quality of non-photorealistic art.
2 Method - non-realistic art processing of images in modern product design
The difficulty of non-photorealistic image processing in modern product design mainly lies in the non-photorealistic processing of real color images, which can be divided into two parts: scientific data visualization and artistic style processing. Visualization of scientific data seeks to highlight important information while ignoring secondary information, and information that cannot be represented by real images is highlighted. Art style processing focuses on artistic expression. This is not to replace the artist’s work, but to increase their efficiency. Through the analysis of the non-photorealistic images after the artist’s non-photorealistic processing, it can be seen that the non-photorealistic processing mainly has the following important features: (1) the transition of regional colors in the image is incoherent, meaning that relatively it is not smooth; (2) the outline of the area is highlighted; (3) the color of the area is relatively uniform and bright; and (4) many details inside the area are eliminated. Compared with the real color image, we can find that to realize the automatic non-photorealistic processing of the computer, we must deal with the above four characteristics of the non-photorealistic image. In simple terms, non-photorealistic processing aims to eliminate the details of the image area, and the area color is processed uniformly, while highlighting the edge contour of the area. However, the real color image has more details in the general area. With the increase in the resolution of the camera, the fineness of the image is higher. This objective problem brings difficulties to the non-realistic processing, and the camera works according to the ambient light. Different adjustments will be made, and the difference in light is directly related to the distribution of gray values for each channel of the RGB image, which brings difficulties in unifying the non-realistic processing style. In general, the objective problems with the non-photorealistic processing of real color images using computer technology are mainly about the development of camera imaging technology and the diversification of the imaging environment. The goal of this method is to solve this problem without introducing human involvement. Under the premise of the circumstances, fast and effective non-photorealistic processing of real images is accomplished.
The non-realistic art image processing method in modern product design proposed in this paper is shown in Fig. 1. First, the RGB color image is converted into a Lab color image, and gradient filtering is performed on the CS channel to obtain an edge gradient map. Then grayscale quantization is performed on the CS channel and the quantized CS channels are combined. The original image is classified into blocks, and appropriate numbers of NRAP measurements are made according to different block characteristics. Subsequently, different nested block optimization class tree estimators are designed for the blocks with different characteristics, and the image blocks are linearly restored. Finally, image blocks are merged into an entire image.
The image x of each pixel N = Io × Ir is divided into n blocks of size B × B, the column vector form of the i block is denoted as:
xi, i = 1, ⋯, n, n = N/B2,and then the same measurement matrix ΦB is used to measure xi, and the observation value vector yi is obtained, the length of which is denoted as MB(≪B2):
The formula ΦB is the random Gaussian matrix of MB × B2, so the M NRAP observations of the original image are obtained. For the whole image, the total measurement matrix Φ is the following block diagonal matrix, and the n diagonal elements are all ΦB:
From Eq. (2), we can see that in the BNRAP measurement, there is no need to store a matrix B × B, Φ but only a matrix block MB × B2, ΦB to be stored in. Obviously, when it is small, the required memory is small and can be realized quickly.
Natural images contain certain local features: flatness, edges, and textures. Integrally graded tree processing is performed on the images and each feature can be identified by these characteristics. A reasonable approach is to make a small number of measurements on flat blocks. For edges and texture blocks, relatively more measurements are made. According to this situation, the image block variance is defined as in the formula in Eq. (3):
In the formula pij, the gray value of pixel of the i block and the j block are expressed, dmin and dmax are represented respectively for the minimum and maximum values of the variance in all the blocks. In this way, the image block classification criteria can be obtained:
T1 and T2 are the thresholds set for the classification, according to experience, we can set the thresholds as T1 = 0.1 and T2 = 0.3. As shown in Fig. 2b, the 512 × 512 Mandrill map is classified by 16 × 16 inner size, white blocks are texture blocks, the gray block is an edge block and the black block is a flat block. By comparing with the original image of Fig. 2a, it can be seen that the hair area with complex texture is all white, the edge area is mostly a gray block, and for a flat face in the area, black blocks are more concentrated, so it is reasonable to use block variance as a classification criterion.
Set fixed measurement rate for r, flat block measurement rate r1, edge block measurement rate r2, texture block measurement rate r3, image total block number N, flat block number N1, edge number N2, and texture block number N3, then:
among them r1 = R r2 = 3R r3 = 5R˳
This means we finish with:
After calculating the value R, we can determine r1, r2, and r3, due to the influence of setting the weight, the flat block measurement rate is low, the texture block measurement rate is high, and the edge block measurement rate is moderate.
In this case, measurement matrices adapted to different types of blocks can be constructed according to different measurement rates. To achieve small storage of measurement matrices and to maintain strong correlation with the height of the reference matrix Ψ, orthogonal Gaussian matrices are used as measurement matrices, and the measurement matrix construction steps are as follows:
-
(1)
Generate a random matrix B2 × B2, Θ, where each element obeys a standard Gaussian distribution.
-
(2)
Using the Schmidt normalization method, the column vectors of the random matrix Θ are orthogonal to each other.
-
(3)
From the measurement rates of the flat, edge, and texture blocks, determine their number of measurements M1, M2 and M3 respectively.
-
(4)
Randomly pick out the M1, M2, and M3 respectively from the matrix Θ, and generate the measurement matrix for the flat, edge, and texture blocks \( {\varPhi}_B^{(1)} \), \( {\varPhi}_B^{(2)} \) and \( {\varPhi}_B^{(3)} \).
Among them, the matrix ΘΩ(|Ω| = m) composed of arbitrary lines m, when satisfying the condition m ≥ C ⋅ S(lgB4)4, is irrelevant to any fixed orthogonal reference matrix Ψ with a high probability of maintaining the height, where S is the sparsity of the coefficient under the image block Ψ, and B4 is the total number of pixels of the image block. The number C is a smaller constant, so that as long as M1, M2, and M3 are sufficiently large, the measurement matrix \( {\varPhi}_B^{(1)} \), \( {\varPhi}_B^{(2)} \), \( {\varPhi}_B^{(3)} \) of the flat, edge, and texture blocks can be guaranteed, and it is not strongly correlated with the reference matrix Ψ to ensure that the image processing algorithm can converge effectively. It can also be seen that only the measurement matrix Θ can be constructed to generate flat, edge, and texture blocks \( {\varPhi}_B^{(1)} \), \( {\varPhi}_B^{(2)} \) and \( {\varPhi}_B^{(3)} \) so the low memory of the measurement matrix is still guaranteed.
Traditional image processing methods all use nonlinear optimization methods to recover images. This image processing process involves a large number of iterative processes, which results in high computational complexity and is not suitable for real-time, non-photorealistic art processing tasks in modern real-time product design. As an initial solution to the image processing algorithm in modern product design, the optimization hierarchy tree has achieved good results, as shown in Fig. 3.
The image block data x is treated as a random vector, and the NRAP measurement data y are obtained by x using a measurement matrix ΦB for linear projection:
Since ΦB is generated by randomly selecting several rows through orthogonal Gaussian random matrices Θ, it has the restricted isometry property (RIP), and there is correlation between the pixels of the block x. Therefore Rxx ≠ 0, according to ΦB, RIP, and it is known from formula (11), that is Rxy ≠ 0, there is correlation between x and y. Therefore, the part related to the image block x can be estimated from the NRAP measurement data y by the principle of optimal estimation.
Setting y after the transformation H, we obtain linear estimate \( \widehat{x} \) and estimated error for:
The criterion for selecting a linear operator H is to minimize the mean square value of the estimation error, expressed as:
Reeis seeking for the derivation from H, and if we let it be equal to 0, we can solve the requirementsHof the embedded block optimization hierarchical tree algorithm.
From the union of the Eqs. (10) and (14), the following can be obtained:
Among them, the statistical autocorrelation function matrix among the image blocks pixels is Rxx:
The elements of this matrix can be estimated by Eq. (17):
where (n1, m1) is the spatial position of the pixel xi, (n2, m2) is the spatial position of the pixel xj, and δ is the Euclidean distance to xi and xj, and the correlation coefficient ρ is between 0.9 and 1.0.
Since the level characteristics of each image block are not the same, for example, the degree of correlation between pixels in a flat block is much larger than that of a texture block, the values ρ of the blocks with different characteristics should be different. According to this situation, the value of the flat block ρ is taken as 0.997, and the edge block ρ is taken as 0.993 and the texture block value ρ is taken as 0.990. Figure 4 shows the comparison of the autocorrelation function values of the flat, edge, and texture blocks sampled from multiple images and the autocorrelation function values estimated by using the above values ρ in Eq. (16). In the figure, the abscissa is the coordinate value of the pixel on the diagonal of the image block, and the ordinate is the autocorrelation function value of the pixel in the upper left corner of the image block x (1, 1) and the other pixels in the diagonal line. The estimated value as can be seen by the actual value deviation is not large, so the value of ρ should be reasonable.
Once the nested block optimization class tree solver Hopt is found and each image block is only a simple matrix-vector inner product, an estimate of the original image block can be obtained. The image is divided into n non-overlapped image blocks; only the n sub-matrix-vector inner product operation can be used to complete the image processing of the entire image. The total amount of calculation requires only M × B2 sub-multiplication and M × B2 sub-adding, which is much smaller than the image processing method in the traditional product design.
3 Experimental simulation results and discussions
In order to verify the superiority and effectiveness of the non-photorealistic processing method proposed in this paper for real color images, a non-photorealistic processing simulation experiment for color images was carried out. Two people using different brands of equipment and different imaging resolutions took a picture of a person scene and an architectural figure, which are 800 × 600 and 800 × 600 respectively. The reason for selecting different imaging devices, different scenes, and different image sizes is to illustrate the generality of the algorithm in this paper. The non-photorealistic processing of these two color images is shown in Fig. 5 as the following.
The non-photorealistic effects of the portrait images of famous celebrities on the Internet are also very good, as shown in Fig. 6. To further explain the superiority of the proposed algorithm, we compared it with the non-photorealistic processing method based on the scan line proposed by previous researchers. These two algorithms are used to perform non-photorealistic processing on the person scene graphs in Fig. 5a. The results are shown in Fig. 7. The average color-filling algorithm based on scan lines in the conventional method obtains boundary points by first performing image segmentation and performs average color filling on the regions between two boundary points in the same row in line order. From Fig. 6, it can be seen that the conventional method is prone to color area jumps and false stripes when it is not processed by realism. This is caused by the shortcomings of the algorithm itself, and the main reason is the edge break. This algorithm can well overcome this point, and it can produce very non-photorealistic effects for the scene maps, architectural drawings and character images captured by different imaging devices. This indicates that the algorithm in the image is not true to the color image. And it is of certain application value on the sense processing.
According to this algorithm, the original image of Sakura, such as in Fig. 8, is processed by non-realistic art, and the result of different degree of processing is obtained by modifying the mask size to adjust the probability and statistical area.
Figure 9 is the effect map of the original image based on the domain pixel probability statistical filtering processing. It is the effect of the filter mask size processing using 5 × 5, 9 × 9, 15 × 15, and 25 × 25. It can be seen that after this algorithm is used, the simulation of the stroke effect is more obvious; the larger the mask size of the filter is, the color block is more condensed. The more obvious the effect is, the greater the scope, and the boundary between scenes is blurred. This method uses the appropriate filter mask size to deal with the original image. It can better simulate the characteristics of the artistic style, such as the Impressionist school, especially in the works of painting, water color, especially oil painting and watercolor, which has weakened the volume sense of the scene and strengthened the color factors. Light and shade and lines form a sense of space, but form the space with the cool and warm colors. In the whole algorithm research process, there are some shortcomings to be further improved: the picture quality and clarity of the input image have a certain influence on the end result and we cannot achieve the best and fixed invariance of each simulation effect. It needs the user to adjust the corresponding parameters or preprocess the picture for different original images. The selection of color space has a certain influence on the simulation results. It is also the main task for further experiment and exploration to choose a reasonable and appropriate color space according to the characteristics of the original picture. The final simulation effect is still different from the real works of art, lacking the “randomness” in the real art works and the individual style characteristics of the artists.
We extracted one 512 × 512 grayscale image from the following five databases (Debbie, Lena, Cameraman, Peppers and Mandrill) to test the image non-photorealistic art processing method based on the inline block optimization hierarchical tree algorithm proposed in this paper. The test images have different degrees of detail. The Debbie diagram contains a large number of flat areas, and the edges and textures are relatively simple. The Lena and Cameraman diagrams also contain large flat areas, but the edges and textures are more abundant than Debbie diagrams; in the Peppers diagram there are a large number of edges and flat areas with less texture detail, and the Mandrill diagram contains a large number of hair textures, with fewer edges and flat areas. The image processing algorithm compared to this algorithm is: BNRAP-SPL image processing using directional transforms. BNRAP-SPL image processing using directional transforms, such as NRAP-SPL-DDWT and NRAP-SPL-CT algorithms; processing of all the images in modern product design is unrealistic (processing time for 512 × 512 images is up to 30 h). To reduce the reconstruction time, image processing strategy in the embedded product modern design is adopted, which is named as BNRAP-TV. The image processing strategy in the embedded product modern design is named BNRAP-TV; the image processing algorithm GPSR is used in the modern product design based on gradient projection, because of the GPSR image processing speed. The fast random Fourier envelope (SFE) measures the entire image and then uses the GPSR algorithm for image processing to name it SFE-GPSR. The size of the embedded block size, B, is taken as 32, based on experience. The NRAP measurement rate (M/N) is taken as 0.5. Since the measurement matrix has randomness, the quality and speed of the reconstructed image will change accordingly, so the peak signal-to-noise ratio (PSNR) value and the reconstruction time are independently averaged for five tests.
The PSNR values of the images processed by the aforementioned algorithms are listed in Table 1. It can be seen that for Debbie, Cameraman and Lena that contain many flat areas, no matter at low or high measurement rate, the proposed algorithm has higher PSNR value in image processing than other algorithms. The PSNR values of image processing images are higher than other algorithms, especially for Debbie images, whereby the PSNR value of image processing images can reach 36.78 dB at a low measurement rate of 0.1, but for images with high edges and textures such as Peppers and Mandrill, The proposed algorithm is not superior to other algorithms: for the Peppers graph, the image processing performance is slightly lower than for other algorithms except the SFE-GPSR algorithm; for the Mandrill graph, the image processing image quality is the worst of all algorithms (Fig. 10). Compared with the optimal recovery quality, it is 1 to ~ 3 dB lower, which is mainly due to the fact that the proposed algorithm does not work well for restoring texture blocks, and there are two reasons for the poor recovery of texture blocks: (1) measurements and (2) the estimated value of the autocorrelation function matrix of the texture block deviate significantly from the real situation (as can be seen from Fig. 4c), by increasing the number of texture block measurements and adjusting the value of the estimated autocorrelation function matrix. To a certain extent, the situation is alleviated. Figure 5 shows the Cameraman images processed by various algorithms when the NRAP measurement rate is 0.2. It can be seen from the subjective comparison that the proposed algorithm has a low measurement rate for images with uncomplicated textures. When the image is restored, there will not be many blockings, and the subjective visual quality is good.
Table 2 lists the time consumed by various algorithms to process 512 × 512 Cameraman images. The hardware platform for running the algorithm is a 2.20 GHz dual-core central processing unit (CPU) computer, and the software platform is a 32-bit Windows 7 operating system and Matlab 7.6 simulation experiment. It can be clearly seen that the image processing time of the algorithm presented in this paper is much lower than other algorithms, and the average time is only 2 s, so it is very suitable for real-time non-photorealistic art processing tasks in modern product design.
4 Conclusion
This paper proposes an image processing method based on the embedded block optimization hierarchical tree algorithm for modern product design. This method introduces the embedded block optimization hierarchical tree algorithm in the modern product design and classifies the image blocks according to the different image blocks to estimate the statistical autocorrelation function matrix, and then carries out non-photorealistic art processing for each image block in the product design. In the modern product design, an adaptive inline block Gaussian random measurement is used. At the decoding end, the statistical self is estimated according to the level characteristics of different blocks. The statistical autocorrelation function matrix is estimated, which is used to further construct the embedded block optimization tree algorithm for each image block. This algorithm breaks through the image processing in the traditional nonlinear modern product design, and greatly shortens the non-realistic art processing time of images. It is especially suitable for real-time non-photorealistic art processing tasks in real-time modern product design, and the experimental results also show that the measurement algorithm of this modern product design does not affect the image processing quality due to the reduction of the computational complexity of the image with uncomplicated texture details. It is still superior to the popular image processing algorithms in modern product design.
Abbreviations
- NRAP:
-
Non-realistic art processing
- PSNR:
-
Peak signal-to-noise ratio
- RIP:
-
Restricted isometry property
- SFE:
-
Fast random Fourier envelope
References
LC Chan, KM Au, KM Yu, Fem–ct integrated design for multiscale damage analysis of hydroformed magnesium-based alloy tubular product. NDT&E Int 56(2), 38–47 (2013)
C Kwong, A methodology of integrating affective design with defining engineering specifications for product design. Int. J. Prod. Res. 53(8), 2472–2488 (2015)
MA Prawira Negara, Design and implementation of roaster control system using image processing. IEEE Trans. Signal Process. 62(21), 5677–5689 (2013)
L Strawderman, Y Huang, Designing product feature upgrades: the role of user processing and design change. Int. J. Ind. Ergon. 42(5), 435–442 (2012)
J Chen, LLT Chan, YC Cheng, Gaussian process regression based optimal design of combustion systems using flame images. Appl. Energy 111(4), 153–160 (2013)
EJ Candes, J Romberg, Robust signal recovery from incomplete observations (IEEE International Conference on Image Processing, Atlanta, 2006), pp. 1281–1284
S Paris, SW Hasinoff, J Kautz, Local laplacian filters: edge-aware image processing with a laplacian pyramid. Commun. ACM 58(3), 81–91 (2015)
J Ragan, Decoupling algorithms from schedules for easy optimization of image processing pipelines. ACM Trans Graph 31(4), 13–15 (2012)
C Tsiotsios, M Petrou, On the choice of the parameters for anisotropic diffusion in image processing. Pattern Recogn. 46(5), 1369–1381 (2013)
CA Deledalle, L Denis, G Poggi, F Tupin, L Verdoliva, Exploiting patch similarity for Sar image processing: the nonlocal paradigm. IEEE Signal Process. Mag. 31(4), 69–78 (2014)
Availability of data and materials
We can provide the data.
Author information
Authors and Affiliations
Contributions
Xu works all things about this paper. The author read and approved the final manuscript.
Corresponding author
Ethics declarations
Authors’ information
Xu Xunxin, Graduated from Xi’an Academy of Fine Arts in 2014 and is a lecturer in the school of clothing and art design, Xi’an Polytechnic University. The main research direction is product art design.
Ethics approval and consent to participate
Approved.
Consent for publication
Approved.
Competing interests
The author declares that he has no competing interests.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Xu, X. Imagery non-photorealistic art processing in modern product design relying on an inline block optimization hierarchy tree algorithm. J Image Video Proc. 2018, 71 (2018). https://doi.org/10.1186/s13640-018-0299-1
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s13640-018-0299-1