Skip to main content

Gradient-based global features for seam carving

Abstract

We propose a gradient-based global feature and its application to seam carving. We focus on areas, rather than points and lines, to be assigned as important elements for expressing the rough location of salient objects in an image. The proposed feature is calculated with a low computational load based on gray-scale intensity. The superior performance of the proposed gradient-based global feature, as compared to state-of-the-art salient features for seam carving, is demonstrated experimentally.

1 Introduction

The goal of image retargeting is to resize an image with a different aspect ratio while preserving salient objects. Image retargeting is attracting increased attention in a wide variety of fields related to digital imaging and display [1]. Seam carving is an approach to image retargeting that has greater flexibility than cropping and resampling and holds a simple algorithm besides [2, 3]. Although several retargeting methods have been developed, such as mesh-deformation-based techniques [46] and some combinations of multiple methods [7], seam carving still remains an effective tool due to the simplicity and the scalability. Seam carving has expanded in applications, such as for video sequences [8], stereo images [9], image compression [10, 11], tamper detection [12, 13], and particular images having high image detail concentration [14].

In seam carving, an eight-connected path of pixels from top to bottom (or from left to right), referred to as a seam, with the minimum energy is deleted or inserted one by one. That is, pixels with higher energy remain untouched, which requires assigning higher energy to the pixels of important elements appropriately. In the original seam carving, gradient magnitude was used as the energy function lest seams should traverse the edges [2]. The problem of seam carving is to determine the seam that should be deleted or duplicated by the accumulative energy. That is, even if a seam passes through distinct edges, if the seam passes through smoother area, such as silhouette, skin, and solid-color clothes especially in dark color, the accumulative energy of the seam can be lower than that of other seams, which results in the distortion of important objects. Several methods have been addressed this problem by newly defining an importance or saliency map. Achanta and Süsstrunk proposed a saliency map based on global colors and intensity contrast [15], which is improved by combining with gradient magnitude [16]. Conger et al. proposed the use of points prohibited from being included in seams in addition to gradient magnitude [17]. These are very simple and effective to a certain extent. By contrast, some methods pursue complex human recognition. Liu et al. constructed various salient features, including multi-scale contrast, center-surround histogram, and color spatial distribution through learning [18]. Goferman et al. proposed context-aware saliency based on four basic principles of human visual attention [19]. These are computationally expensive.

In the present paper, we address the problem whereby important objects expressed as smoother areas are distorted in seam carving. We propose gradient-based global features of images for an importance map, which are obtained by gray-scale intensity with low computational cost. The proposed method is based on the simple idea that, rather than points and lines, areas that include salient objects in an image should be given importance and is motivated by the histogram of oriented gradients (HoG) [20] and the scale-invariant feature transform (SIFT) [21, 22], which are well-established feature descriptors based on the weighted histogram of a gradient. However, the proposed method does not require making histograms like HoG and SIFT but using the inverse variance of weighted gradient orientation. Besides, the proposed method is not limited to color images like [15, 16]. We demonstrate the effectiveness of the proposed method compared to the conventional methods. A shorter version of this paper is presented in [23]. The present paper is refined with additional examples from the shorter version and includes objective comparisons.

The remainder of the present paper is organized as follows. In Section 2, the gradient local feature and seam carving are explained. In Section 3, related research is described. In Section 4, gradient-based global features are proposed. In Section 5, simulations are conducted in order to demonstrate the effectiveness and appropriateness of the proposed features. Finally, Section 6 concludes the paper.

2 Preliminaries

The proposed feature is based on gradient, but different from conventional gradient-based local features, and designed to seam carving. In this section, gradient, gradient-based local features, and seam carving are described.

2.1 Gradient

Let I(n 1,n 2), n 1=0,1,…,N 1−1, n 2=0,1,…,N 2−1, be an image of size N 1×N 2, where n 1 and n 2 are vertical and horizontal indices, respectively.

The gradient of I(n 1,n 2) is defined as

$$\begin{array}{*{20}l} \nabla I(n_{1}, n_{2}) = \left[ \begin{array}{c} \nabla_{\mathrm{n}_{1}} I(n_{1}, n_{2}) \\ \nabla_{\mathrm{n}_{2}} I(n_{1}, n_{2}) \end{array} \right] \end{array} $$
(1)

where

$$\begin{array}{*{20}l} \nabla_{\mathrm{n}_{1}} I(n_{1}, n_{2}) = I(n_{1}, n_{2}) - I(n_{1}-1, n_{2}), \end{array} $$
(2)
$$\begin{array}{*{20}l} \nabla_{\mathrm{n}_{2}} I(n_{1}, n_{2}) = I(n_{1}, n_{2}) - I(n_{1}, n_{2}-1). \end{array} $$
(3)

The gradient magnitude is defined as

$$\begin{array}{*{20}l} || \nabla I(n_{1}, n_{2}) || = \sqrt{\left\{\nabla_{\mathrm{n}_{1}} I(n_{1}, n_{2})\right\}^{2} + \left\{\nabla_{\mathrm{n}_{2}} I(n_{1}, n_{2})\right\}^{2}} \end{array} $$
(4)

which is often approximated by the absolute value as

$$\begin{array}{*{20}l} | \nabla I(n_{1}, n_{2}) | = | \nabla_{\mathrm{n}_{1}} I(n_{1}, n_{2}) | + | \nabla_{\mathrm{n}_{2}} I(n_{1}, n_{2}) |. \end{array} $$
(5)

The gradient orientation is defined as

$$\begin{array}{*{20}l} \theta(n_{1}, n_{2}) = \tan^{-1} \frac{\nabla_{\mathrm{n}_{1}} I(n_{1}, n_{2}) }{\nabla_{\mathrm{n}_{2}} I(n_{1}, n_{2})}. \end{array} $$
(6)

where θ(n 1,n 2) is in the range [−π/2,π/2].

2.2 Vector gradient-based local features

Histogram of oriented gradients (HoG) and scale-invariant feature transform (SIFT) are referred to as the gradient-based local features, the central idea of which is the use of a weighted gradient orientation histogram of a local area.

Let us consider a weighted gradient orientation histogram of an area A. The entire range of the gradient orientation [Θ 1,Θ 2] is divided into a total of N b bins of width D=(Θ 2Θ 1)/N b .

The weighted gradient orientation histogram, h(b), for b=1,2,…,N b , is expressed as

$$\begin{array}{*{20}l} h(b) = \sum_{n_{1} \in A} \sum_{n_{2} \in A} w(n_{1}, n_{2}) \delta(c(b), \theta(n_{1}, n_{2})) \end{array} $$
(7)

where w(n 1,n 2) is the weight for which gradient magnitude is used, θ(n 1,n 2) is the gradient orientation in (6), c(b) is the center value of the bth bin, and

$$\begin{array}{*{20}l} \delta(c, \theta) = \left\{ \begin{array}{ll} 1, & c - D/2 \leq \theta < c + D/2 \\ 0, & \text{otherwise} \end{array} \right. \end{array} $$
(8)

which counts the number of orientations that fall within a bin. The vector of the components of this weighted gradient orientation histogram is used as a descriptor.

2.3 Seam carving

Original seam carving established the fundamental procedure, in which an image is resized according to the path of pixel locations, referred to as the seam. Based on seam energy, the seams are identified for removal or insertion.

A seam is an eight-connected path from top to bottom (or from left to right) that contains only one pixel per row (or column). A vertical seam, \(s_{n_{1}}\), for n 1=1,2,…,N 1, is a sequence of coordinates defined as

$$\begin{array}{*{20}l} s_{n_{1}} = & \{(n_{1}, T(n_{1}))\} \end{array} $$
(9)
$$\begin{array}{*{20}l} \text{s.t} \quad &\forall n_{1} \quad | T(n_{1}) - T(n_{1}-1) | \leq 1 \end{array} $$
(10)

where T(·) is a mapping from the ordinate, n 1, to the location of the abscissa [1,2,…,N 2]. A horizontal seam, \(s_{n_{2}}\), is obtained in the same manner.

The seam energy, E, is the total pixel energy of constitutive pixels and is obtained as

$$\begin{array}{*{20}l} E = \sum_{n_{i} = 1}^{N_{i}} e(s_{n_{i}}) \end{array} $$
(11)

where i=1 or 2, and \(e(s_{n_{i}})\) denotes the pixel energy e(n 1,n 2) at location (n 1,n 2), which is given as

$$\begin{array}{*{20}l} e(n_{1}, n_{2}) = | \nabla I(n_{1}, n_{2}) |. \end{array} $$
(12)

The seam energy of all possible seams in an image is calculated, and the seam with the minimum energy is identified for removal or insertion.

Practically speaking, the seam with the minimum energy is effectively found by dynamic programming, in which the accumulative cost matrix M(n 1,n 2) for all possible seam connections is calculated as

$$ \begin{aligned} M(n_{1}, n_{2}) &= e(n_{1}, n_{2}) + \text{min}(M(n_{1}-1, n_{2}-1),\\ &\quad M(n_{1}-1, n_{2}), M(n_{1}-1, n_{2}+1)) \end{aligned} $$
(13)

or

$$ {{ \begin{aligned} {}M(n_{1}, n_{2})& = e(n_{1}, n_{2}) + \text{min} (M(n_{1} - 1, n_{2}-1) + C_{L}(n_{1}, n_{2}), \\ &M(n_{1}\,-\,1, n_{2}) \,+\, C_{U}(n_{1}\!, n_{2}), M(n_{1} \,-\, 1, n_{2} \,+\, 1) \,+\, C_{R}(n_{1}, n_{2})) \end{aligned}}} $$
(14)

where

$$\begin{array}{*{20}l} C_{L}(n_{1}, n_{2}) &= |I(n_{1}, n_{2}+1)-I(n_{1}, n_{2}-1)| \\ &\quad+ |I(n_{1}-1, n_{2}) - I(n_{1}, n_{2}-1)| \end{array} $$
(15)
$$\begin{array}{*{20}l} C_{U}(n_{1}, n_{2}) &= |I(n_{1}, n_{2}+1)-I(n_{1}, n_{2}-1)| \end{array} $$
(16)
$$\begin{array}{*{20}l} C_{R}(n_{1}, n_{2}) &= |I(n_{1}, n_{2}+1)-I(n_{1}, n_{2}-1)| \\ &\quad+ |I(n_{1}-1, n_{2})-I(n_{1}, n_{2}+1)|. \end{array} $$
(17)

Note that (13) and (14) are referred to as the backward energy and forward energy criteria, respectively [3].

When important objects are monotonic against a background, the objects might be distorted because the deleted or inserted seam is chosen according to the minimum cumulative energy. The goal of the present study is to determine how to assign pixel energy so as to suppress the distortion of important objects with monotonic intensity for seam carving.

3 Related research

In this section, we briefly describe related research, Euclidian distance of the average Lab vector, prohibited points, and context-aware saliency. These three are designed to address the problem mentioned above.

3.1 Euclidian distance of the average Lab vector

Color can be a main feature for an object. The Euclidian distance between the average color and the color at a pixel in Lab color space makes a saliency map for detecting important objects [15].

The saliency map, e Lab(n 1,n 2), is obtained as

$$\begin{array}{*{20}l} e_{\text{Lab}} (n_{1}, n_{2}) = || \mathbf{I}_{\mu} - \mathbf{I}_{\text{blur}}(n_{1}, n_{2}) || \end{array} $$
(18)

where I μ is average Lab vector of the Lab color space, I blur(n 1,n 2) represents the Lab vector at location (n 1,n 2) of the Gaussian-blurred version, and ||·|| expresses the Euclidean distance. The pixel energy is expressed by the saliency map as

$$\begin{array}{*{20}l} e(n_{1}, n_{2}) = e_{\text{Lab}} (n_{1}, n_{2}). \end{array} $$
(19)

In balanced energy map [16], the pixel energy is given so that seams may not concentrate in a particular low energy area as

$$\begin{array}{*{20}l} e(n_{1}, n_{2}) = | \nabla I(n_{1}, n_{2}) | + \alpha e_{\text{Lab}} (n_{1}, n_{2}) \end{array} $$
(20)

where α is a constant in the range [0, 1] and e Lab is normalized into the range [0, 1].

3.2 Prohibited points

The seam criteria are restricted by points prohibited from being included in seams [17]. Mono-directional gradient magnitude is used as a secondary energy map for selecting candidates of pixels for the prohibited points. As the secondary energy map, the horizontal gradient magnitude is used for width reduction, and the vertical gradient magnitude, for hight reduction. This is based on the observation that the deletion of the vertical seam passing through a high-horizontal-energy area, in which most of gradient vectors of large magnitude are oriented in the horizontal direction as illustrated in Fig. 1 a, causes more distortion than that passing through a high-vertical-energy area as illustrated in Fig. 1 b.

Fig. 1
figure 1

Images and gradient vectors in high-horizontal-energy and high-vertical-energy areas. In a high-horizontal-energy areas, most of the gradient vectors of large magnitude are oriented in the horizontal direction; conversely, in b high-vertical-energy area, most of them, in the vertical direction

The prohibited point is determined from a secondary energy map as

$$\begin{array}{*{20}l} P(n_{1}, n_{2}) = \left \{ \begin{array}{cc} 1, & | \nabla_{\mathrm{n}_{i}} I(n_{1}, n_{2}) | \geq \eta \\ 0, & | \nabla_{\mathrm{n}_{i}} I(n_{1}, n_{2}) | < \eta \end{array} \right. \end{array} $$
(21)

where η is the threshold and P(n 1,n 2)=1 means that the pixel at the location (n 1,n 2) must not be included in seams. The threshold η is chosen manually, and otherwise the threshold is given as

$$\begin{array}{*{20}l} \eta = \kappa \sigma \end{array} $$
(22)

where σ is the standard deviation of the \(\phantom {\dot {i}\!}| \nabla _{\mathrm {n}_{i}} I(n_{1}, n_{2}) |\) and κ is set to approximately 2.

3.3 Context-aware saliency

The concept of context-aware saliency is that salient regions should contain the parts of the background that convey the context as well as the prominent objects [19]. This concept is based on four basic principles of human visual attention and is supported by psychological evidence. The algorithm is computationally expensive due to the extensive use of the K-nearest neighbor algorithm.

Figure 2 shows typical and state-of-the-art salient features. The saliency results for gradient magnitude (Fig. 2 b) and prohibited points [17] (Fig. 2 d) focus on points. The saliency results for Euclidean distance of the average Lab vector [15] (Fig. 2 c) and context-aware saliency [19] (Fig. 2 e) focus on areas.

Fig. 2
figure 2

Salient features. a Original image. b Gradient magnitude. c Euclidean distance of the average Lab vector. d Prohibited points. b and d categorize point or line features, whereas c and e categorize area features

4 Proposed method

We herein propose the inverse variance of weighted gradient orientation as a local feature and the superposition of the local features as a global feature for seam carving. In the following, for the sake of brevity, we consider width reduction.

4.1 Scalar gradient-based local feature

Taking a macroscopic view, we observe that a gradient vector having a large magnitude is concentrated on the edges of an object, whereas such vectors in a local area are orientated in roughly the same direction, as shown in Fig. 3. The use of weighted gradient orientation of a uniformly divided area as a local feature is motivated by HoG. However, unlike HoG, the local feature is not a vector but rather a scalar. As such, we no longer need to make a histogram of each area.

Fig. 3
figure 3

Gradient vectors indicated as arrows at each pixel. a Original image of size 32 × 32. b Gradient vectors. Gradient vectors of large magnitude are concentrated on the edges of objects and are orientated in roughly the same direction

Let n 1 and n 2 be the vertical and horizontal locations, respectively, in block B. The mean \(\bar {\theta }_{B}\) of the weighted gradient orientation of B is given as

$$\begin{array}{*{20}l} \bar{\theta}_{B} &= \frac{\sum \sum_{n_{1}, n_{2} \in B} w(n_{1}, n_{2})\theta(n_{1}, n_{2})}{\sum \sum_{n_{1}, n_{2} \in B}w(n_{1}, n_{2})} \end{array} $$
(23)

where θ(n 1,n 2) is the gradient orientation in (6) and w(n 1,n 2) is the weight using the gradient magnitude:

$$\begin{array}{*{20}l} w(n_{1}, n_{2}) = |\nabla I(n_{1}, n_{2}) |. \end{array} $$
(24)

The variance \({\sigma ^{2}_{B}}\) of the weighted gradient orientation of B is defined as follows:

$$\begin{array}{*{20}l} {\sigma^{2}_{B}} &= \frac{\sum \sum_{n_{1}, n_{2} \in B}w(n_{1}, n_{2})(\theta(n_{1}, n_{2}) - \bar{\theta}_{B})^{2} }{ \sum \sum_{n_{1}, n_{2} \in B} w(n_{1}, n_{2})}. \end{array} $$
(25)

The scalar gradient-based local feature of B is given by

$$\begin{array}{*{20}l} D_{B} = \frac{1}{{\sigma^{2}_{B}} + \epsilon} \end{array} $$
(26)

where ε is small value.

A large inverse variance indicates that an edge exists in the block, whereas a small inverse variance indicates that either no edges or multiple edges exist, as shown in Fig. 4.

Fig. 4
figure 4

Inverse variance of the weighted gradient orientation of areas. b and c are gradient vectors on the upper and lower areas, respectively, denoted by square in a. The inverse variance is 1.15 and 15.29, respectively

The value of the inverse tangent in the process obtaining the gradient orientation makes the desirable effect that a pixel in a high-horizontal-energy area is set greater value than that in a high-vertical-energy area. In a high-horizontal-energy area, the gradient orientation θ(n 1,n 2) in (6) is concentrated around 0 and π, as shown in Fig.1 a, at which the inverse tangent is 0, resulting in a large inverse variance. On the other hand, in a high-vertical-energy area, θ(n 1,n 2) is concentrated around −π/2 and π/2, as shown in Fig. 1 b, at which the inverse tangent is −π/2 and π/2, respectively, resulting in a small inverse variance. Thus, seams including the pixel in high-horizontal-energy area are suppressed from being identified for removal.

4.2 Gradient-based global feature

The gradient-based global feature consists of the scalar gradient-based local feature in multiple layers. The size of a layer of the gradient-based global feature is the same as the original image. The value at a location in the layer is assigned to the scalar gradient-based local feature the location belongs to. An image is divided into nonoverlapping pixel blocks, the size of which is fixed in a single layer, but differs in each layer. A layer with smaller block size exposes features at a smaller scale. The variation of the block size yields a multi-scale structure.

Let \({B_{i}^{l}}\) be the ith block in the lth layer (l=1,,L). The lth layer, g l(n 1,n 2), of the gradient-based global feature is expressed using (26) as

$$\begin{array}{*{20}l} g^{l}(n_{1}, n_{2}) = D_{{B^{l}_{i}}} \end{array} $$
(27)

where the pixel location (n 1,n 2) falls within \({B^{l}_{i}}\). The gradient-based global feature, G(n 1,n 2), is given by accumulating a total of L layers as

$$\begin{array}{*{20}l} G(n_{1}, n_{2}) = \sum_{l = 1}^{L} &g^{l}(n_{1}, n_{2}). \end{array} $$
(28)

The number of layers and the block size of each layer are manually chosen.

Figure 5 illustrates an example of the gradient-based global feature, where the number of layers is two, the block sizes of which are 2×2 and 4×4. In this example, the value of the gradient-based global feature at the location in black is the sum of \(D_{{B_{6}^{1}}}\) and \(D_{{B_{1}^{2}}}\).

Fig. 5
figure 5

Concept of gradient-based global feature. Each layer of an image is divided into nonoverlapping pixel blocks. The gradient-based local features of blocks are accumulated over pixels

Figure 6 shows an example of the gradient-based global feature and the corresponding seams that are identified for removal (shown in red). The gradient-based global feature represents the rough locations of salient objects in an image, and the seam locations obtained using multiple layers are better than those obtained using a single layer.

Fig. 6
figure 6

An example of the gradient-based global feature and the corresponding seams that are identified for removal. a The first layer of block size 8 × 8. b The seams that are identified by a. c The second layer of block size 16 × 16. d The seams that are identified by c. e The gradient-based global feature consisting of a and c. f The seams that are identified by e

The seam carving procedure using gradient-based global features is summarized as follows.

  1. 1.

    Calculate the gradient-based global feature G(n 1,n 2) as e(n 1,n 2) according to (28)

  2. 2.

    Generate accumulative cost matrix according to (13) or (14)

  3. 3.

    Update I(n 1,n 2) by deleting/inserting the seam with minimum energy from the image

  4. 4.

    Update G(n 1,n 2) by deleting/inserting the seam with minimum energy from the gradient-based global feature

  5. 5.

    Repeat steps 2 to 4 until the desired size is obtained

5 Simulations

We compared the proposed feature as an importance measure for seam carving with typical and state-of-the-art features for seam carving. We measured the ratio of deformation on specific zones and computational time in order to show the effectiveness of the proposed method.

5.1 Deformation on specific zones

The results obtained using the proposed feature were compared with the results obtained using the gradient magnitude (GM), the balanced energy map (BEM) [16], the prohibited points (PP) [17], and the context-aware saliency (CAS) [19] (We used the MATLAB implementation from [24]). Twenty-four-bit 320×480 pixel color images having important objects with monotonic intensity were used for evaluation, which were obtained from the Berkeley Segmentation Dataset [25]. Except for the BEM and the CAS, the color image was converted into a gray-scale image beforehand by

$$ \begin{aligned} g(n_{1}, n_{2}) &= 0.2126 \, r(n_{1}, n_{2}) + 0.7152 \, g(n_{1}, n_{2}) \\&+ 0.0722 \, b(n_{1}, n_{2}) \end{aligned} $$
(29)

where r(n 1,n 2), g(n 1,n 2), and b(n 1,n 2) denote the color values (0 to 255) for the red, green, and blue channels, respectively, for each pixel. A total of 144 pixels (30 %) of the horizontal width were removed. In the BEM, λ was set to 0.2. In the PP, η was calculated according to (22) where σ was set to the standard deviation of \(\phantom {\dot {i}\!}| \nabla _{\mathrm {n}_{2}} I(n_{1}, n_{2})|\) and κ was set to 2. In the proposed feature, the number of layers was two, the block sizes of which were 8 × 8 and 16 × 16.

Figures 7, 8, 9, 10, and 11 show the resulting images obtained by the GM, the PP, the BEM, the CAS, and the proposed feature using (14). Although simple and efficient to some content, the results obtained using the PP shown in Figs. 7, 9, 10, and 11 were better than the results using GM but were worse than the results using the BEM, the CAS, and the proposed feature. The results using the BEM shown in Figs. 7, 8, and 10 were better than those using the GM and the PP. However, the BEM depends on colors of important objects. The results using the BEM shown in Figs. 9 and 11 were worse than the results using the CAS and the proposed feature. Although the CAS provides almost better results, the results shown in Fig. 11 were not good compared to results using proposed feature.

Fig. 7
figure 7

Comparison of salient features for seam carving with 70 % of the width of the original image “children”. a Original image. b GM. c PP. d BEM. e CAS. f Proposed

Fig. 8
figure 8

Comparison of salient features for seam carving with 70 % of the width of the original image “golf”. a Original image. b GM. c PP. d BEM. e CAS. f Proposed

Fig. 9
figure 9

Comparison of salient features for seam carving with 70 % of the width of the original image “man”. a Original image. b GM. c PP. d BEM. e CAS. f Proposed

Fig. 10
figure 10

Comparison of salient features for seam carving with 70 % of the width of the original image “people”. a Original image. b GM. c PP. d BEM. e CAS. f Proposed

Fig. 11
figure 11

Comparison of salient features for seam carving with 70 % of the width of the original image “priest”. a Original image. b GM. c PP. d BEM. e CAS. f Proposed

We measured the deformation on specific zones in an objective way. The measurement of deformation is defined by the ratio of the number of deleted pixels on specific zones to the number of pixels on specific zones. Figure 12 shows ground truth for specific zones of each image. Tables 1 and 2 show the percentage of deformation on specific zones with backward energy criteria and forward energy criteria, respectively. We observed that the results using the proposed feature are almost the least deformation on the specific zones.

Fig. 12
figure 12

Ground truth. a Children. b Golf. c Man. d People. e Priest

Table 1 Percentage of deformation on specific zones with backward energy criteria
Table 2 Percentage of deformation on specific zones with forward energy criteria

5.2 Computational complexity

We evaluated computational time for generating a map. The CPU was 2.4 GHz Intel Core i7 and the software was MATLAB. We used the author’s distribution including a MEX file (C, C++, or Fortran subroutines loaded and executed by MATLAB) for the CAS.

Table 3 shows the computational time for generating a map. The fastest method was the PP and the second was the BEM. The speed of the proposed method was no match for that of these methods but it was much less than one second. The CAS took much time for generation due to K-nearest neighbor algorithm although a MEX file was used.

Table 3 Computational time (sec.) for generating a map

5.3 Comparison to seam carving using depth-assisted saliency map

The use of depth map in seam carving is originally designed for a pair of stereo images to keep 3D structure of the scene [9] and later adopted for a single image to avoid distortion in important objects [26, 27]. Although not all images have their depth maps, the strategy focused on depth map offers a new way of determining the important objects.

We compared the proposed method with the state-of-the-art seam carving using depth assisted saliency map (DAS). In the DAS, the pixel energy is assigned by combining three different components, gradient, a depth map, and a saliency map [27]. The saliency map was generated by a hierarchical saliency method [28] (We used the executable from [29]). The twenty four-bit color stereo images, “Diana” of size 427×427 and “snowman” of size 364×453, and their depth maps were used for evaluation, which were obtained from [30]. Fifty percent of width of the images was reduced. We also performed seam carving using the GM, the PP, the BEM, and the CAS. The parameters of the PP, the BEM, and the proposed feature were the same as those in Section 5.1.

Figures 13 and 14 show the resulting images for “Diana” and “snowman”, respectively. The image obtained using the DAS for “Diana” (Fig. 13 f) was superior to the images obtained using the other methods. However, the DAS does not place importance on background regions, which causes the distortion of them. In the image using the DAS for “snowman” (Fig. 14 f), the distortion of the slope of the roof behind the snowman was noticeable compared to the images using the other methods. Meanwhile, the image using the proposed method for “Diana” (Fig. 13 g) was worse than the image using the DAS but obviously better than the images using the GM, the PP, and the BEM. The image using the proposed method for “snowman” (Fig. 14 g) was superior to the images using the DAS, in which the proposed method avoids the distortion of the slope of the roof while maintaining the steering wheel and its supporting bar. The computational time of the proposed feature with MATLAB implementation on 2.4 GHz Core i7 was 0.1597 [s] and 0.1259 [s] for “Diana” and “snowman,” respectively. Considering the computational time to generate a hierarchical saliency map, which takes average 0.28 [s] for an image of size 400 × 300 with C++ implementation on 3.4 GHz CPU [28], besides the time to generate a depth map and to process them, the proposed method is much faster than the DAS.

Fig. 13
figure 13

Comparison of salient features with 50 % of the width of the original image “Diana”. a Original image. b GM. c PP. d BEM. e CAS. f DAS. g Proposed

Fig. 14
figure 14

Comparison of salient features with 50 % of the width of the original image “snowman”. a Original image. b GM. c PP. d BEM. e CAS. f DAS. g Proposed

6 Conclusions

We have proposed gradient-based global features to suppress the distortion of salient objects with monotonic intensity in seam carving. Based on the observation that the gradient vector on a partial edge faces approximately the same direction in the local area, we have proposed the inverse variance of weighted gradient orientation as a local feature and the superposition of local features in multiple layers as a global feature. The proposed feature is gray-scale intensity-based and is calculated once for seam carving. We have demonstrated the effectiveness of the proposed features by comparing them to typical and state-of-the-art salient features for seam carving.

References

  1. M Rubinstein, D Gutierrez, O Sorkine, A Shamir, A comparative study of image retargeting. ACM Trans. Graph.29(6)Article No. 160, (2010) doi:s10.1145/1882261.1866186.

  2. S Avidan, A Shamir, Seam carving for content-aware image resizing. ACM Trans. Graph.26(3), 1–10 (2007).

    Article  Google Scholar 

  3. M Rubinstein, A Shamir, S Avidan, Improved seam carving for video retargeting. ACM Trans. Graph.27(3), 1–9 (2008).

    Article  Google Scholar 

  4. Y-S Wang, C-L Tai, O Sorkine, T-Y Lee, Optimized scale-and-stretch for image resizing. ACM Trans. Graph.27(5), 1181–1188 (2008).

    Article  Google Scholar 

  5. Y Guo, F Liu, J Shi, Z-H Zhou, M Gleicher, Image retargeting using mesh parametrization. IEEE Trans. Multimed.11(5), 856–867 (2009).

    Article  Google Scholar 

  6. S-S Lin, I-C Yeh, C-H Lin, T-Y Lee, Patch-based image warping for content-aware retargeting. IEEE Trans. Multimed.15(2), 359–368 (2013).

    Article  Google Scholar 

  7. M Rubinstein, A Shamir, S Avidan, Multi-operator media retargeting. ACM Trans. Graph.28(3), 1–11 (2009).

    Article  Google Scholar 

  8. B Yan, K Sun, L Liu, Matching-area-based seam carving for video retargeting. IEEE Trans. Circ. Syst. Technol.23(2), 302–310 (2013).

    Article  Google Scholar 

  9. T Basha, Y Moses, S Avidan, Stereo seam carving: a geometrically consistent approach. IEEE Trans. Pattern Anal. Mach. Intell.35(10), 1816–1823 (2013).

    Article  Google Scholar 

  10. Y Tanaka, M Hasegawa, in IEEE International Conference on Image Processing. Improved image concentration for artifact-free image dilution and its application to image coding (IEEEHong Kong, 2010), pp. 1225–1228.

    Google Scholar 

  11. E Salma, JP Josh Kumar, Efficient image compression based on seam carving for arbitrary resolution display devices. Int. J. Comput. Appl.68(4), 37–40 (2013).

    Google Scholar 

  12. T Yamasaki, T Matsunami, K Aizawa, in Sixth International Conference on Intelligent Information Hiding and Multimedia Signal Processing. Detecting resized jpeg images by analyzing high frequency elements in DCT coefficients (Darmstadt, 2010), pp. 567–570.

  13. TK Wattanachote, WL Shir, WL Chang, HH Chang, Tamper detection of jpeg image due to seam modifications. IEEE Trans. Inform. Forensics Secur.10(12), 2477–2491 (2015).

    Article  Google Scholar 

  14. M Frankovich, A Wong, Enhanced seam carving via integration of energy gradient functionals. IEEE Signal Process. Lett.18(6), 375–378 (2011).

    Article  Google Scholar 

  15. R Achanta, S Süsstrunk, in IEEE International Conference on Image Processing. Saliency detection for content-aware image resizing (IEEECairo, 2009), pp. 1005–1008.

    Google Scholar 

  16. J Chen, L Miao, X Liu, Image re-targeting with balanced energy map and foreground constraint. World Wide Web. 14(3), 281–292 (2011).

    Article  Google Scholar 

  17. DD Conge, M Kumar, RL Miller, L Jiebo, H Radha, in IEEE Workshop on Signal Processing Systems. Improved seam carving for image resizing (IEEESan Francisco, 2010), pp. 345–349.

    Google Scholar 

  18. T Liu, J Sun, N-N Zheng, X Tang, H-Y Shum, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Learning to detect a salient object, (2007), pp. 1–8.

  19. S Goferman, L Zelnik-Manor, A Tal, Context-aware saliency detection. IEEE Trans. Pattern Anal. Mach. Intell.34(10), 1915–1926 (2012).

    Article  Google Scholar 

  20. N Dalal, B Triggs, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Histograms of oriented gradients for human detection (IEEESan Diego, 2005), pp. 886–893.

    Google Scholar 

  21. DG Lowe, in IEEE International Conference on Computer Vision (ICCV). Object recognition from local scale-invariant features (Kerkyra, 1999), pp. 1150–1157.

  22. DG Lowe, Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. (IJCV). 60(2), 91–110 (2004).

    Article  Google Scholar 

  23. I Ito, in APSIPA Annual Summit and Conference. Gradient-based global features and its application to image retargeting (Hollywood, 2012), pp. 1–4.

  24. Context-aware Saliency Matlab Implementation. http://cgm.technion.ac.il/Computer-Graphics-Multimedia/Software/Saliency/Saliency.html. Accessed 2010.

  25. D Martin, C Fowlkes, D Tal, J Malik, in 8th Int. Conf. Computer Vision. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics (IEEEVancouver, 2001), pp. 416–423.

    Google Scholar 

  26. J Shen, D Wang, X Li, Depth-aware image seam carving. IEEE Trans. Cybern.43(5), 1453–1461 (2013).

    Article  Google Scholar 

  27. F Shafieyan, N Karimi, B Mirmahbo, S Samavi, S Shirani, in IEEE International Conference on Image Processing. Image seam carving using depth assisted saliency map (IEEEParis, 2014), pp. 1155–1159.

    Google Scholar 

  28. Q Yan, L Xu, J Shi, J Jia, in IEEE Conference on Vision and Pattern Recognition. Hierarchical saliency detection (IEEEPortland, 2013), pp. 1155–1162.

    Google Scholar 

  29. Hierarchical Saliency Executable. http://www.cse.cuhk.edu.hk/leojia/projects/hsaliency/. Accessed Apr 2016.

  30. Stereo Seam Carving. http://people.csail.mit.edu/talidekel/StereoSC.html. Accessed Apr 2016.

Download references

Competing interests

The authors declare that they have no competing interests.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Izumi Ito.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ito, I. Gradient-based global features for seam carving. J Image Video Proc. 2016, 27 (2016). https://doi.org/10.1186/s13640-016-0130-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13640-016-0130-9

Keywords