Integrated visual security management for optimal condition explorations in resource-constrained systems

2.1 Configuration framework to support the JVCE

For the video compression, we utilize the JM H.264 reference software (version 18.4) (http://iphome.hhi.de/suehring/tml). Further, we need to select and implement several video encryption techniques among various ones. The video encryption methods of the early stage were designed to protect the whole video data by using the conventional encryption algorithms such as the DES (Data Encryption Standard) [30] and the AES (Advanced Encryption Standard) [31]. Those techniques have high computational complexity [32–34]. Moreover, they do not consider the format of the compressed bitstream defined in the standard video codecs. If the format is broken, direct manipulations on the bitstream such as parsing, transcoding, and decoding cannot be supported [12, 35, 36]. This is not appropriate for various video services such as the subscription-based television, the video-on-demand, and the video surveillance systems. In order to keep the format compliance while reducing the computational complexity of the algorithm, several video encryption techniques are introduced considering the characteristics of the video compression process and the video data [6–15]. They protect partial data within the compression process through just simple operations. Among them, we implement several methods [6–8] in the JM software as presented in Fig. 1.

In more detail, the utilized encryption methods protect the coefficients and the motion vectors that are spatial and temporal information, respectively. They pseudo-randomly flip sign bits of those data [6, 8], xor them with pseudo-random numbers [7, 8], pseudo-randomly shuffle them with each other [7], and apply possible combinations of those three methods. We do not shuffle the coefficients since we need to re-calculate the number of zero coefficients prior to each non-zero coefficient, incurring the extra computations. If we do not re-calculate that numbers after shuffling the coefficients, the format of the bitstream could be messed up. In case of the coefficient protection, the encryption methods are applied to the DC (direct current) coefficient only, the AC (alternating current) coefficient only, and both of them in the luminance channel. Further, we change possible range of pseudo-random numbers for the XOR method from 0 to small and large maximum in case of both coefficients and motion vectors. When using the pseudo-random numbers from the larger range, the video is distorted by the XOR method more heavily than the case of using the smaller range. This can increase the size of the compressed bitstream so that it should be discussed in the design space exploration in our results.

2.2 Design space exploration framework of the JVCE

As shown in Fig. 2, we complete the experimental framework for the design space exploration of the JVCE using the raspberry pi 3 model B board (https://www.raspberrypi.org/products/raspberry-pi-3-model-b/), the Monsoon power monitor (http://www.msoon.com/LabEquipment/PowerMonitor), and the desktop. The Monsoon power monitor accurately estimates the energy consumption of the JVCE running in the Ubuntu 16.04 LTS (https://wiki.ubuntu.com/ARM/RaspberryPi) which is specified for the raspberry board. After conducting the JVCE on the board, we can get the compression efficiency returned from the H.264 codec software. Then, we decompress the encrypted bitstream without the video decryption in the desktop to obtain the encrypted videos. Further, the encrypted videos are estimated by the implemented visual security metrics to represent the visual security. For the visual security evaluation, we exploit (https://ece.uwaterloo.ca/texttildelowz70wang/research/ssim/) [37] and (http://live.ece.utexas.edu/research/Quality) for the SSIM (Structural SIMilarity) [38], the LEG (Local Edge Gradients) [39], and the VIF (Visual Information Fidelity) [40], respectively, and implement other evaluation methods presented in the following section. Finally, we can examine the design space consisting of every possible set of parameters of the JVCE utilized in this study in terms of the visual security, the compression efficiency, and the energy efficiency.

For the design space exploration, we adjust various parameters of the video encryption explained in previous subsection such as the kind of data, the type of the coefficient, and the number of the applied encryption techniques. Additionally, we also adjust the video compression parameters such as the GOP size and the QP (quantization parameter) level. The number of motion vectors to be protected by the encryption methods depends on the GOP size leading to different amount of the overheads. In our experiments, we vary the GOP size as 1, 5, 10, 30, and 300. In addition, we examine the trend while changing the QP levels as 15, 20, and 25, since the QP significantly incurs the impact on the compression efficiency and the energy efficiency. Finally, we protect the Akiyo, Coastguard, and Foreman QCIF videos of 300 frames while varying all the parameters in the JVCE as described so far.

3.1 Objective visual quality/security evaluation metrics

In the image processing studies, many wonder how much an image is perceptually changed after the compression, the encryption, and the transmission on the error-prone network, as such. The perceptual change typically represents the quality degradation or the protection power. In this case, we can compare the processed image with the reference one in terms of the human eye’s perception such that “the processed image is heavily distorted, slightly twisted, or quite unseen”. Based on those subjective expressions, it is difficult to objectively determine whether one image is more perceptually distorted than other ones in the way that many people agree with. Thus, for the clear comparison, we need to quantify the degree of the visual difference between the target images as close as possible to the human eye’s perception and also the way many people agree with. Accordingly, various evaluation metrics have been proposed to estimate how much the target images perceptually differ as the numerical levels. These metrics are typically called the visual quality metrics. Further, the term visual security metric is often used to indicate the evaluation method to show the amount of the perceptual distortion introduced by the video encryption techniques.

The PSNR (peak signal-to-noise ratio) is one of the most popular metrics in the signal processing to represent the difference between two images. However, the PSNR has often been argued that it is inaccurate to show the evaluation results similar to the ones from the human eye’s perception [41–43]. For the better representation of the visual difference between the images than the PSNR, there have been several approaches including the SSIM and the LFBVS (local feature-based visual security) [44]. The SSIM captures the visual difference between the images better than the PSNR [45]. However, it has been turned out that it shows inaccurate evaluation for the heavily blurred images [46]. The LFBVS [44] has been designed to more effectively represent the visual difference between the original video and the encrypted video as compared to the conventional metrics including the PSNR and the SSIM. The LFBVS appropriately estimates the visual distortion introduced by the video encryption techniques. However, it shows the inaccurate evaluation on several encrypted videos since it does not consider the distortion of the temporal information such as the movements of the objects in the consecutive frames [22]. The LEG appears a good correlation with human eye’s perception, and further, the VIF outperforms various metrics mentioned above including LEG [47]. However, they still lack in considering temporal information of consecutive images in the same video.

In order to compensate the inaccurate estimation cases of the conventional metrics, we designed the STVSM (spatio-temporal visual security metric) in our previous research [22]. The STVSM considers the distortions of both the spatial and the temporal information such as the luminance coefficient and the motion vector, respectively. Finally, the STVSM effectively captures not only the spatial distortion of each frame but also the temporal distortion of the consecutive frames of the encrypted video [22]. Accordingly, we employ the STVSM for the design space exploration of the JVCE regarding the visual security. Before the exploration, we comprehensively demonstrate the STVSM’s evaluation as compared to several conventional metrics in the following subsections.

3.2 Visual security evaluation setup

For the encrypted videos by using those four scenarios, we clearly demonstrate how much the objective evaluation correlates with the subjective evaluation that is the human eye’s perception. To obtain the subjective evaluation results as the reference, we conduct a survey for 66 university students. In the survey, the participants watch both the original QCIF videos and the distorted videos by the encryption schemes defined in Table 3. To control the lighting condition which could affect the visual assessments of the participants, we use the same laptop under the same contrast and brightness for all the survey. For each encrypted video, the participants score the degree of how much the video is perceptually distorted as compared to the reference one from 1 to 5. The scores indicate “Highly recognized (1)”, “Partially recognized (2)”, “Medium (3)”, “Slightly distorted (4)”, and “Totally distorted (5)”, respectively. The higher score indicates the higher visual security. Based on the survey, the MOS (mean opinion score) is calculated by averaging the subjective scores for each encrypted video. We do not consider the outlier detection when averaging the MOS, since most of the scores typically appear in the similar range. The averaged MOS scores are compared with the objective evaluation results.

For the objective evaluation, we exploit the luminance channel data since the change of the luminance information is more sensitive to human eyes than that of the chrominance information [48]. For the SSIM, we use the default setting in the [38] so that the MSSIM (mean SSIM) is estimated per each frame by averaging the SSIM of the local windows within the frame. The PSNR, the MSSIM, the LEG, and the VIF represent the visual difference of each image in the video so that we utilize the average of each metric score of total frames in the encrypted video. From 0 to 1, lower MSSIM, higher LFBVS, lower LEG, lower VIF, and higher STVSM of the encrypted video indicate that the video is more visually protected than others. Further, the PSNR is ranged from 0 to infinite where closer value to 0 implies that the encrypted video is more different from the reference one than others. In order to claim that higher values of the estimation results from four metrics indicate higher visual security, we examine the 1/PSNR, 1-MSSIM, 1-LEG, and 1-VIF instead of PSNR, MSSIM, LEG, and VIF, respectively.

3.3 Objective visual security evaluation results as compared to subjective ones

Figures 3 and 4 show the objective scores and the subjective scores for all the scenarios in Table 3. For all the graphs in both Figs. 3 and 4, the evaluation scores are represented in the abstract manner such as from low to high since the range of the scores differ based on the type of metric, target video, and parameters. Moreover, the examined range of the objective scores is appropriately scaled to clearly show the trend as compared to that of the subjective scores. Note that even if several objective scores present the trend not corresponding to that of the subjective scores such as the zigzag pattern in Fig. 3 (g), they reveal several inaccurate cases, not total failure. From left to right on the x-axes of all the graphs, the evaluation results in one scenario are arranged in increasing order of MOS.

In the results, the evaluation scores by using the STVSM show more similar trend to that of the subjective score MOS as presented in Fig. 3(p), (q), and (r) as compared to other metrics. Moreover, we examine the visual security evaluation results additionally adjusting the range option for the xor encryption operation in Fig. 4. Similarly, it is demonstrated that the STVSM estimates the perceptual distortion more close to the subjective score than other metrics. Figure 5 shows the sample frames of each encrypted video corresponding to the same order of the x-axes of the graphs in Fig. 4. From Fig. 5(a) to (p), it is demonstrated that the visual contents of the video become more noisy and blurred.

In the scenario Extended-1, the visual distortion of the encrypted videos differs highly depending on whether the motion vectors are protected or not and which encryption techniques are applied to motion vectors. For example, we compare two encrypted videos by encryption schemes P and Q which have different motion encryption methods in the scenario Extended-1: xor_enc(large_r){AC}shuffle_enc{MV} and xor_enc (large_r){AC}shuffle_enc{sign_enc{xor_enc(large_r) {MV}}}, respectively. Figure 6 presents four sample frames from the original Foreman video and two encrypted versions by P and Q. Both P and Q distort the original video in noisy manner, but we can see the perceptual difference between the encrypted videos due to the different motion vector encryption techniques. Note that not all the frames of the encrypted video by P leak the visual information more than those by Q. However, it is true that the encrypted video by P has more frames to leak perceptual clues to guess the original contents shown in Fig. 6(e), (f), (g), and (h). The subjective scores which are MOS scores in Table 5 also demonstrate the clear difference between two encrypted videos. However, the 1/PSNR, the LFBVS, and the 1-LEG cannot determine that which one of P and Q perceptually distorts the video more than the other. Although the scores of the 1-MSSIM and the 1-VIF show the perceptual difference of both videos, the difference is only 3% which is ambiguous and even negligible as compared to the clear difference as 12% in case of the STVSM.

Encryption	1/PSNR	1-MSSIM	LFBVS	1-LEG	1-VIF	STVSM	MOS
P	0.122	0.463	0.666	0.880	0.934	0.561	3.530
Q	0.122	0.476	0.668	0.886	0.963	0.631	4.667

One of the main reasons of the inaccurate evaluation cases of the conventional visual quality metrics is lacking in the consideration of the temporal distortion of the encrypted videos. The temporal information such as the trajectories of the moving objects can be leaked when playing the consecutive frames even if each frame is well encrypted. Unfortunately, the conventional visual quality metrics estimate the visual security of the encrypted video only in the spatial manner which considers the visual distortion between the target image and the reference one. On the other hand, the STVSM compares the degree of the visual distortion not only between the encrypted frame and the reference one but also between the consecutive frames in the encrypted video and the corresponding ones in the reference video. Based on thorough investigation in this section, the STVSM works as a good measure to objectively represent the perceptual distortion of the encrypted videos as compared to several conventional metrics. Therefore, we utilize the STVSM to examine the design space in terms of the visual security in the following section.

4.1 Bitrate and energy-bound visual security

In the design space of the JVCE, exploiting various sets of parameters result in different amount of the visual security, the compression efficiency, and the energy efficiency. The values of three indices have been turned out that they have tradeoff relationships between each other. Accordingly, it is difficult to find one optimal set of parameters among various ones to achieve all those indices to the best. Therefore, we like to find the interesting solution in the design space to maximize the visual security while satisfying the energy budget, the visual security threshold, and the compression bitrate threshold. In order to achieve that, we extend the design space exploration of the JVCE in terms of the visual security and the energy efficiency [29] by additionally considering the compression efficiency under several budgets. The compression efficiency should be considered since reducing the size of the video is the main goal of the video compression. Although the JVCE with one specific set of parameters achieves the maximum visual security under the budgets, the set could significantly degrade the compression efficiency.

4.2 Heuristic exploration algorithm for the design space of the JVCE

According to various parameters of the JVCE such as the GOP size and the number of utilized encryption methods, the design space of the JVCE contains lots of parameter sets to be examined. Therefore, an efficient exploration algorithm for the design space of the JVCE is required. In this work, we design a simple but effective heuristic algorithm to relieve the exploration overheads. Based on the experimental results of the JVCE in this work, we empirically design the exploration algorithm DSE in Algorithm 1. The DSE finds the interesting set of parameters among various ones to obtain the maximum BEVS while satisfying the compression bitrate threshold B _thres, the energy budget E _bud, and the visual security threshold VS _thres. The key idea of the DSE is that if we can classify the sets of parameters into groups resulting in different range of the visual security without overlapping, it is possible to save the exploration overheads by examining the group of higher visual security range prior to other groups. By doing so, the DSE does not examine all the sets at one time but classifies the total sets into several groups based on the visual security (line 2). The criteria to determine the number of groups and the branch points of those groups without overlapping should be specific to target applications. Further, the DSE searches for one set providing the maximum visual security group by group under the given requirements (lines 3–6). In this case, we examine the group of the range of higher visual security prior to the groups of the range of lower visual security. Therefore, if we find one set to obtain the maximum visual security within one group under the requirements, the DSE immediately finishes the exploration returning it as the solution (line 7).

Algorithm 2 represents the example of the customized exploration algorithm customDSE which is designed for the design space of the JVCE in this paper. In Algorithm 2, we exploit three observations from our experimental results; (1) the trend of the visual security differs according to the GOP size and the QP level, (2) the xor encryption significantly enhances the visual security, and (3) the groups determined by the options for the xor encryption and the range of the pseudo-random numbers have different ranges of the visual security which are not overlapped. Based on these observations, we decide to show the heuristic exploration algorithm under the fixed QP level since considering the combination of the GOP size and the QP level at once makes the algorithm too much complex and long to represent in this paper. Moreover, the GOP size is more related with the amount of the data to be encrypted by the schemes that we implement as compared to the QP level. Therefore, we conduct the design space exploration under the fixed QP level 25 while adjusting the GOP size utilized in this study (lines 4–34). Initially, we examine the sets of parameters under the GOP size 1 (lines 5–17). Under the GOP size 1, we classify the sets of parameters into three groups based on the xor encryption and the range of the pseudo-random numbers (lines 6–8). The group1, group2, and group3 are in the decreasing order of the visual security and not overlapped. From group1 to group3, if we find a set of parameters to satisfy all the requirements COND consisting of E _bud, B _thres, and VS _thres, the set is assigned to set _gop==1 (lines 9–17). For the GOP size 5, 10, and 30, we commonly classify the sets of parameters into two groups based on the option for the xor encryption (lines 19–21). And then the similar exploration of the GOP size 1 is conducted (lines 22–27). In case of the GOP size 300, there are no intuitive ways to group the sets of parameters so that we use the exhaustive search (lines 28–32). After the exploration, we have the candidate sets of parameters (line 35). Finally, we find the interesting set of parameters denoted as set _result to have the maximum BEVS among the candidates of five GOP sizes and the customDSE returns it (line 36 and 37).