Oriented relative fuzzy connectedness: theory, algorithms, and its applications in hybrid image segmentation methods

Abstract

Anatomical structures and tissues are often hard to be segmented in medical images due to their poorly defined boundaries, i.e., low contrast in relation to other nearby false boundaries. The specification of the boundary polarity can help alleviate a part of this problem. In this work, we discuss how to incorporate this property in the relative fuzzy connectedness (RFC) framework. We include a theoretical proof of the optimality of the new algorithm, named oriented relative fuzzy connectedness (ORFC), in terms of an oriented energy function subject to the seed constraints, and show its usage to devise powerful hybrid image segmentation methods. The methods are evaluated using medical images of MRI and CT of the human brain and thoracic studies.

1 Introduction

Manipulating large amounts of data efficiently with high performance is today a complex task investigated by various scientific communities, as well by private sector corporations and government entities. Within this context, computational methods that make use of graphs as a basic element of study have played a key role in getting innovative solutions in various fields of knowledge, in particular in problem areas of computer vision and information visualization. Recent examples of applications that employ graph analysis in their processing pipelines are easily found in the literature such as: segmentation and classification of images via large-scale graphs [1,2], rearrangement and removal of overlaps in visual layouts, visualization and high-dimensional data clustering [3,4], among others. Thus, the modern theory of graphs is seen today as an indispensable tool to explore, analyze, and process large volumes of information, especially when it comes to digital images and high-dimensional data visualization, in view of its strong theoretical and mathematical support [5].

In this work, we explore graphs by modeling neighborhood relationships of picture elements from digital images for the purposes of image segmentation, such as to extract an object from a background, by assigning different labels to its picture elements. This labelling process is useful for many applications, such as medical and biological image analysis and digital matting, being a well-pursued topic in image processing and computer vision.

One important class of graph-based image segmentation methods comprises interactive seed-based methods, where the user provides a partial labelling of the image by placing hard region-based constraints (known as seeds). After that, the seed’s labels are propagated to all unlabeled regions by following some optimum criterion, such that a complete labeled image is constructed. This class encloses many of the most prominent methods for general purpose segmentation, which are usually easier to extend to multi-dimensional images, including frameworks, such as watershed from markers [6,7], random walks [8], fuzzy connectedness [9,10], graph cuts (GC) [11], distance cut [12], image foresting transform [13], and grow cut [14]. The study of the relations among different frameworks, including theoretical and empirical comparisons, has a vast literature [15-19], which allowed many algorithms to be described in a unified manner according to a common framework, which we refer to as generalized GC (GGC) [18,20]. Within this framework, in the discrete labelling case, there are two important classes of energy formulations, the ε ₁- and ε _∞ -minimization problems (and so, the associated algorithms), as discussed in [20].

In this work, we are interested in fast seed-based methods to efficiently deal with large amounts of data but which must also be versatile enough to support the inclusion of high-level, soft constraints. A soft constraint imposes a penalty on certain labelling assignments rather than prohibiting them. The penalty values allow the customization of the segmentation to different objects according to their expected high-level features (shape constraints, boundary polarity), which can be learned from a training dataset.

The most time-efficient seed-based approaches of the GGC framework are the ones that fall within the ε _∞ -minimization problem, which have linear time implementations O(N) with respect to the image size N [17], while the run time for the ε ₁-minimization problem is O(N ^2.5) for sparse graphs [21]. Recently, some methods from the ε _∞ -minimization family were extended to support the boundary polarity constraint, by exploring directed weighted graphs, leading to the method named oriented image foresting transform (OIFT) [22,23]. While the introduction of combinatorial graphs with directed edges on other frameworks increases considerably the complexity of the problem [24], the OIFT still runs in linear time. The boundary orientation/polarity helps to resolve between very similar nearby boundary segments with opposite transitions (dark to bright/bright to dark). The usage of directed weighted graphs also allows the incorporation of shape constraints as demonstrated in [25].

In this work, we discuss how to incorporate this orientation information, by exploring digraphs, in another member of the ε _∞ -minimization family, a region-based approach called relative fuzzy connectedness (RFC) [26]. RFC is an important method, which presents some nice theoretical properties, such as the robustness with respect to the seed choice [26]. The regions where the seeds are free to move without affecting the segmentation are called in some works as the cores [10]. In RFC, the cores for each seed coincide with its corresponding delineated regions by RFC. The cores of RFC are key elements in theoretical analysis to support effective semi-automatic correction (i.e., to fix a poor automatic segmentation in an interactive tool [27,28]), by finding a suitable set of seeds that assembles a given segmentation [10]. The RFC also has the advantage of producing a low false positive rate, which allows it to be combined with other methods in powerful hybrid approaches [29,30].

A short version of this work was published in a conference paper [31]. Here, the proposed method, named oriented relative fuzzy connectedness (ORFC), is presented in more details, including experiments involving large three-dimensional datasets, and showing the running time curves. We also extend the hybrid approach [29] to directed weighted graphs, incorporating the boundary polarity by combining the strengths of oriented relative fuzzy connectedness and graph cut. The novel hybrid approach is more robust than the original graph cut with respect to the seed choice (thus, avoiding ‘shrinking problem’ of GC), and it also outperforms the previous hybrid method [29] and OIFT, with running times close to linear.

Section 2 explains the basic concepts on image graphs and introduces the terminology and notation to be used throughout the text. Section 3 shows the original RFC. Section 4 presents the related oriented image foresting transform. The proposed extension of RFC, named oriented relative fuzzy connectedness, is presented in Section 5, and its applications in hybrid image segmentation (ORFC and graph cut) are shown in Section 6. Sections 7 and 8 discuss the experimental results and conclusions.

2 Background

A multi-dimensional and multi-spectral image $\hat {I}$ is a pair $(\mathcal {I},\vec {I})$ where $\mathcal {I}\subset Z^{n}$ is the image domain and $\vec {I}(a)$ assigns a set of m scalars I _i (a), i=1,2,…,m, to each pixel $a\in \mathcal {I}$. The subindex i is removed when m=1.

An image can be interpreted as a weighted digraph G=〈V,E,w〉 whose nodes V are the image pixels in its image domain $\mathcal {I}\subset Z^{n}$ and whose arcs are the ordered pixel pairs 〈a,b〉∈E. For example, one can take E to consist of all pairs of pixels 〈a,b〉 in the Cartesian product $\mathcal {I} \times \mathcal {I}$ such that d(a,b)≤ρ and a≠b, where d(a,b) denotes the Euclidean distance and ρ is a specified constant (e.g., 4-neighborhood, when ρ=1, and 8-neighborhood, when $\rho =\sqrt {2}$, in case of 2D images). The digraph G is symmetric if for any of its arcs 〈a,b〉, the pair 〈b,a〉 is also an arc of G. Each arc 〈a,b〉∈E has a fixed weight w(a,b)≥0, between neighboring pixels, which is ideally designed to have lower values in the boundary transitions of the object of interest (e.g., w(a,b)=K−|I(a)−I(b)|, where K is the greatest difference in image brightness for a single-channel image with values given by I(a)). A symmetric digraph is undirected weighted if w(a,b)=w(b,a) for all 〈a,b〉∈E; otherwise, we have a directed weighted digraph.

The transpose G ^T=〈V,E ^T,w ^T〉 of a weighted digraph G=〈V,E,w〉 is the unique weighted digraph on the same set of vertices V with all of the arcs reversed compared to the orientation of the corresponding arcs in G (i.e., for any of its arcs 〈a,b〉∈E ^T, the pair 〈b,a〉 is an arc of G, and w ^T(a,b)=w(b,a)). A weighted digraph G is symmetric and undirected weighted if G is the same as its transpose.

For a given image graph G=〈V,E,w〉, a path π _a =〈t ₁,t ₂,…,t _n =a〉 is a sequence of adjacent pixels with terminus at a pixel a. A path is trivial when π _a =〈a〉. A path π _b =π _a ·〈a,b〉 indicates the extension of a path π _a by an arc 〈a,b〉. When we want to explicitly indicate the origin of a path, the notation $\pi _{a \rightsquigarrow b}=\langle t_{1} = a,t_{2},\ldots,t_{n} = b \rangle$ may also be used, where a stands for the origin and b for the destination node. More generally, we can use $\pi _{S \rightsquigarrow b}=\langle t_{1},t_{2},\ldots,t_{n} = b \rangle$ to indicate a path with origin restricted to a set S (i.e., t ₁∈S). A digraph is said to be strongly connected if there is a path from every vertex to every other vertex. A connectivity function computes a value f(π _a ) for any path π _a , usually based on arc weights. A path π _a is optimum if f(π _a )≥f(τ _a ) for any other path τ _a in G.

For every weighted digraph G=〈V,E,w〉, consider the space $\tilde {\mathcal {X}}$ of all functions x :V→ [ 0,1], referred to as fuzzy subsets of V, with the value x(a) indicating a degree of membership with which a belongs to the set. The family of all functions $x \in \tilde {\mathcal {X}}$ with the only allowed values of 0 and 1 (i.e., x:V→{0,1}) will be referred to as the family of all hard subsets of V. Each $x\in \mathcal {X}$ is identified with the true subset P={c∈V:x(c)=1} of V. Notice that, in such a case, x is the characteristic function χ _P of P⊂V. We usually restrict the collection of all allowable objects by indicating two disjoint sets, referred to as seeds: $\mathcal {S}_{{o}} \subset V$ indicating the object and $\mathcal {S}_{{b}} \subset V$ indicating the background.

This restricts the collection of allowable outputs of the algorithm to the family $\mathcal {X}(\mathcal {S}_{{o}},\mathcal {S}_{{b}})$ of all $x\in \mathcal {X}$ with x(a)=1 for all $a\in \mathcal {S}_{{o}}$ and x(b)=0 for all $b\in \mathcal {S}_{{b}}$. Note that $\mathcal {X}(\mathcal {S}_{{o}},\mathcal {S}_{{b}})=\{\chi _{P} \colon \mathcal {S}_{{o}}\subset P\subset V\setminus \mathcal {S}_{{b}}\}$.

3 RFC

3.1 The original definition by connectivity functions

Next, we show the original RFC definition as proposed in [32] for undirected weighted graphs. Consider the following connectivity function:

$$\begin{array}{@{}rcl@{}} f^{\mathcal{S}}_{\min}(\langle a\rangle) & = & \left\{ \begin{array}{ll} w_{\text{max}}+1 & \text{if}\, a\in {\mathcal{S}} \\ -\infty & \text{otherwise} \end{array} \right. \\ f^{\mathcal{S}}_{\min}(\pi_{a}\cdot \langle a,b\rangle) & = & \min\{f^{\mathcal{S}}_{\min}(\pi_{a}),w(a,b)\} \end{array} $$

where w _max= max〈a,b〉∈E w(a,b). Two connectivity maps are computed by using two executions of the image foresting transform (IFT) [13]:

$$ V_{{o}}(a) = \max_{\pi_{a}\: \in \:\Pi(G,a)} \{f_{\text{min}}^{\mathcal{S}_{{o}}}(\pi_{a}) \}, $$

((1))

$$ V_{{b}}(a) = \max_{\pi_{a}\: \in \:\Pi(G,a)} \{f_{\text{min}}^{\mathcal{S}_{{b}}}(\pi_{a}) \}, $$

((2))

where Π(G,a) is the set of all possible paths in the graph G with terminus at the node a. The segmentation $A_{\text {RFC}}(\mathcal {S}_{{o}},\mathcal {S}_{{b}})$ of the RFC method is obtained by comparing the two maps of connectivity V _o and V _b , such that each pixel a∈V is labeled as belonging to the object only if V _o (a)>V _b (a) (Figure 1).

$$ A_{\text{RFC}}(\mathcal{S}_{{o}}, \mathcal{S}_{{b}}) = \chi_{O} : O= \{ a \in V \colon V_{{o}}(a) > V_{{b}}(a) \} $$

((3))

Figure 1

Example of RFC following the definition based on paths.(a) The object seed ×, where $f_{\min }^{\mathcal {S}_{{o}}}(\pi _{a})=8,\: f_{\min }^{\mathcal {S}_{{o}}}(\pi _{b})=1,\:f_{\min }^{\mathcal {S}_{{o}}}({\pi _{c}})=6\:$, and $\:f_{\min }^{\mathcal {S}_{{o}}}(\pi _{d})=7$. (b) The background seed (asterisk), where $f_{\min }^{\mathcal {S}_{{b}}}(\pi _{d})=8,\: f_{\min }^{\mathcal {S}_{{b}}}(\pi _{b})=1,\:f_{\text {min}}^{\mathcal {S}_{{b}}}(\pi _{c})=6\:$, and $\:f_{\min }^{\mathcal {S}_{{b}}}(\pi _{a})=7$. (c) Segmentation result via RFC.

3.2 RFC as a ε _∞ -optimizer

The RFC method can also be seen as an optimum cut in the undirected weighted graph according to an appropriate objective function of graph cut, as discussed in [17,19,33].

For q∈ [1,∞] consider the energy functional ${\varepsilon }_{q} : \tilde {\mathcal {X}}\to [\!0,\infty)$, where, for every $x\in \tilde {\mathcal {X}}$, ε _q (x) is defined as the q-norm of the functional $F_{x}\colon E\to {\mathbb R}$, given by the formula F _x (a,b)=w(a,b)|x(a)−x(b)| for 〈a,b〉∈E. That is, ${\varepsilon }_{q}(x)= ||F_{x}||_{q}=\sqrt [q]{\displaystyle \sum _{{\langle } a,b{\rangle }\in E}\left (w(a,b)|x(a)-x(b)|\right)^{q}}$, for q<∞.

Notice that $\lim \limits _{\textit {q}\to \infty }{\varepsilon }_{q}(x)={\varepsilon }_{\infty }(x)$, since q-norms converge, as q→∞, to the ∞-norm.

$$ \varepsilon_{\infty}(x)=||F_{x}||_{\infty}=\max_{{\langle} c,d{\rangle}\in E} w(c,d)|x(c)-x(d)| $$

((4))

Restricting the analysis to only binary solutions $x= \chi _{P} \in {\mathcal {X}}$, we have:

$$ \begin{aligned} \varepsilon_{\infty}(x)&=||F_{x}||_{\infty} = \max_{{\langle} c,d{\rangle}\in E} w(c,d)|x(c)-x(d)|\\&=\max_{{\langle} a,b {\rangle} \in {\mathcal{C}}(x)}w(a,b), \end{aligned} $$

((5))

where ${\mathcal {C}}(x) = \{ {\langle } a,b {\rangle } \in E : x(a) \neq x(b) \}$ is a set of cutting edges.

Let ε _{∞ ↓} be the minimum of the energy ε _∞ (x) over all allowable objects $x\in {\mathcal {X}}({\mathcal {S}}_{{o}},{\mathcal {S}}_{{b}})$, that is, ${\varepsilon }_{\infty \downarrow }=\min \{{\varepsilon }_{\infty }(x) \colon x\in {\mathcal {X}}({\mathcal {S}}_{{o}},{\mathcal {S}}_{{b}})\}$. Any element of ${\mathcal {X}}_{\infty }({\mathcal {S}}_{{o}},{\mathcal {S}}_{{b}})=\{x\in {\mathcal {X}}({\mathcal {S}}_{{o}},{\mathcal {S}}_{{b}}) : {\varepsilon }_{\infty }(x)={\varepsilon }_{\infty \downarrow }\}$ will be referred to as an optimum energy solution of ε _∞ in ${\mathcal {X}}({\mathcal {S}}_{{o}},{\mathcal {S}}_{{b}})$. Any algorithm A that, given a graph and seed sets ${\mathcal {S}}_{{o}}$ and ${\mathcal {S}}_{{b}}$, returns an object, denoted by $A(\mathcal {S}_{{o}},\mathcal {S}_{{b}})$, from $\mathcal {X}_{\infty }(\mathcal {S}_{{o}},\mathcal {S}_{{b}})$ will be referred to as an ε _∞ -minimizing algorithm. The RFC algorithm is an ε _∞ -minimizing algorithm, that is, $A_{\text {RFC}}({\mathcal {S}}_{{o}},\mathcal {S}_{{b}}) \in \mathcal {X}_{\infty }(\mathcal {S}_{{o}}, \mathcal {S}_{{b}})$ [29]. In the case of a single internal seed s ₁ (Figure 2), we have the following alternative definition of RFC based on graph cut:

$$ \begin{aligned} {A}_{\text{RFC}}(\{ s_{1} \},\mathcal{S}_{{b}}) &= \chi_{O} \in \mathcal{X}_{\infty}(\{ s_{1} \},\mathcal{S}_{{b}}) \colon |O|\\&=\min\{ |P| :\chi_{P} \in \mathcal{X}_{\infty}(\{ s_{1} \},\mathcal {S}_{{b}}) \} \end{aligned} $$

((6))

Figure 2

Example of the RFC definition by optimum energy. The three above solutions (a,b,c) have optimum energy ε _∞ (x)=7, but we have that only (c) corresponds to a valid RFC solution according to Equation 6 for the seeds given in Figure 1.

The case of multiple internal seeds is then treated using the following equation:

$$ {A}_{\text{RFC}}({\mathcal{S}}_{{o}},{\mathcal{S}}_{{b}}) = \chi_{O} \colon O = \left[\bigcup_{s_{i} \in {\mathcal{S}}_{{o}}} P : \chi_{P} = A_{\text{RFC}}(\{ s_{i} \},{\mathcal {S}}_{{b}}) \right] $$

((7))

4 OIFT

A directed weighted graph is computed, where w(a,b) is a combination of a regular undirected similarity measure δ(a,b), multiplied by an orientation factor, as follows:

$$\begin{array}{@{}rcl@{}} w(a,b) & = & \left\{ \begin{array}{ll} \delta(a,b)\times (1-\alpha) & \text{if}\, I(a) > I(b) \\ \delta(a,b)\times (1+\alpha) & \text{if}\, I(a) < I(b)\\ \delta(a,b) & \text{otherwise} \end{array} \right. \end{array} $$

((8))

Several different procedures can be adopted for δ(a,b), such as the complement of the absolute value of the difference of image intensities (i.e., δ(a,b)=K−|I(a)−I(b)|), or the affinity functions discussed in [34,35]. Note that we have a directed weighted graph (w(a,b)≠w(b,a)) when α>0.

The oriented image foresting transform is build upon the IFT framework by considering the following path function in a symmetric digraph:

$$f^{S_{1},S_{2}}_{\text{OIFT}}(\langle a\rangle)= \left\{ \begin{array}{ll} w_{\text{max}}+1 & {\text{if}\,\, a\in S_{1} \cup S_{2}}\\ -\infty & \text{otherwise} \end{array} \right. $$

$${\fontsize{8}{6}\begin{aligned} f^{S_{1},S_{2}}_{\text{OIFT}}(\pi_{c\rightsquigarrow a}.\langle a,b\rangle)= \left \{\begin{array}{ll} \min\left\{f^{S_{1},S_{2}}_{\text{OIFT}}(\pi_{c\rightsquigarrow a}),2\cdot w(b,a) \right\} & {\text{if}\,\, c\in S_{1}} \\ \min\left\{f^{S_{1},S_{2}}_{\text{OIFT}}(\pi_{c\rightsquigarrow a}),2\cdot w(a,b)+1 \right\} & \text{otherwise} \\ \end{array} \right. \end{aligned}} $$

OIFT has two versions: $A_{\text {OIFT}}^{in}(S_{{o}},S_{{b}})$ which favors transitions from dark to bright pixels, and $A_{\text {OIFT}}^{\text {out}}(S_{{o}},S_{{b}})$ which has the opposite orientation. $A_{\text {OIFT}}^{\text {in}}(S_{{o}},S_{{b}})$ is obtained by computing one IFT with connectivity function $f_{\text {OIFT}}^{S_{{o}},S_{{b}}}$, and by taking as object pixels the set of pixels that were conquered by paths rooted in S _o . $A_{\text {OIFT}}^{\text {out}}(S_{{o}},S_{{b}})$ is similarly computed, but using $f_{\text {OIFT}}^{S_{{b}},S_{{o}}}$.

One important thing to note is that the function $f_{\text {OIFT}}^{S_{1},S_{2}}$ is a non-smooth connectivity function, as shown in [22]. When a path-value function is not smooth, the IFT will still return a spanning forest, but the paths may not be optimum [13]. However, the optimality of OIFT is still supported by an energy criterion of cut in graphs [22,23].

5 ORFC

Differently from RFC, the definitions of ORFC based on paths and based on cuts in the digraph lead to different results (Figures 3a,b). The different obtained algorithms will be denoted as $A_{\text {ORFC}}^{\text {in},\rightsquigarrow }$ and $A_{\text {ORFC}}^{\text {out},\rightsquigarrow }$ for the path-based definition and and for the cut-based definition.

Figure 3

Example of the different ORFC definitions.(a) ORFC by reverse connectivity functions, with orientation from dark to bright pixels ($A_{\text {ORFC}}^{\text {in},\rightsquigarrow }$). (b) ORFC as a directed cut in the digraph (). (c) The region of seed robustness (core) of.

5.1 ORFC definition by reverse connectivity functions

Based on the previous works [22,23], we consider the following new connectivity function in digraphs:

$$\begin{array}{@{}rcl@{}} f^{\not\parallel\:\mathcal{S}}_{\min}(\langle a\rangle) & = &\left\{ \begin{array}{ll} w_{\text{max}}+1 & {\text{if}\; a\in {\mathcal{S}}} \\ -\infty & \text{otherwise} \end{array} \right. \\ f^{\not\parallel\:\mathcal{S}}_{\min}(\pi_{a}\cdot \langle a,b\rangle) & = & \min\{f^{\not\parallel\mathcal{S}}_{\min}(\pi_{a}),w(b,a)\} \end{array} $$

where 〈b,a〉 is an anti-parallel arc.

Note that $f^{\not \parallel \:\mathcal {S}}_{\min }$ is a smooth function, and therefore, $V_{{o}}^{\not \parallel }$ and $V_{{b}}^{\not \parallel }$ are optimum connectivity maps. These two connectivity maps are generated by executing the IFT with anti-parallel connectivity functions:

$$ V_{o}^{\not\parallel}(a)=\displaystyle\max_{\pi_{a}\in\Pi(G,a)}\{f_{min}^{\not\parallel\:S_{o} }(\pi_{a}) \} $$

((9))

$$V_{b}^{\not\parallel}(a)=\displaystyle\max_{\pi_{a}\in\Pi(G,a)} \{f_{min}^{\not\parallel\:S_{b} }(\pi_{a}) \} $$

((10))

Following the same key idea from [22] (i.e., to consider reversed connectivity functions for one of the seed sets), we have the following natural definition for ORFC: The segmentation $A_{\text {ORFC}}^{\text {out},\rightsquigarrow }(S_{{o}},S_{{b}})$ favoring transitions from bright to dark pixels is obtained by comparing the connectivity maps V _o (a) and $V_{{b}}^{\not \parallel }(a)$, such that each pixel a∈V is labeled as belonging to the object only if $V_{{o}}(a) > V_{{b}}^{\not \parallel }(a)$.

$$ A_{\text{ORFC}}^{\text{out},\rightsquigarrow}(S_{{o}},S_{{b}})=\chi_{O} \colon O=\{ a \in V \colon V_{{o}}(a) > V_{{b}}^{\not\parallel}(a)\} $$

((11))

The segmentation $A_{\text {ORFC}}^{{in},\rightsquigarrow }(S_{{o}},S_{{b}})$ favoring transitions from dark to bright pixels is obtained by comparing the connectivity maps $V_{o}^{\not \parallel }(a)$ and V _b (a), such that each pixel a∈V is labeled as belonging to the object only if $V_{o}^{\not \parallel }(a) > V_{{b}}(a)$.

$$ A_{\text{ORFC}}^{\text{in},\rightsquigarrow}(S_{{o}},S_{{b}})=\chi_{O}: O=\{ a \in V : V_{o}^{\not\parallel}(a) > V_{{b}}(a)\} $$

((12))

Note that although this ORFC version is based on optimum connectivity maps, its practical results have undesirable characteristics, such as the presence of disconnected regions and high false-positive rates, leading to unsatisfactory results (Figure 3a).

5.2 ORFC as a directed cut in the digraph

Given that the previous ORFC definition (Section 5.1) presents undesirable results, in this section, we present an alternative definition supported by a graph cut optimality criterion, which is motivated by the definitions from Section 3.2.

Differently from Section 3.2, in the case of directed graphs, we have two possible sets of cuts (Figure 4):

$$ \mathcal{C}_{\text{out}}(x) = \{ {\langle} a,b {\rangle} \in E : x(a) = 1 \wedge x(b) = 0 \} $$

((13))

Figure 4

The two possible sets of cuts. The inner and outer cuts for a candidate object showing the input and output arcs (a,b).

$$ \mathcal{C}_{\text{in}}(x) = \{ {\langle} a,b {\rangle} \in E \colon x(a) = 0 \wedge x(b) = 1 \} $$

((14))

So we have two possible formulations for the energy functional of the ε _∞ -minimizing problem.

$$ \varepsilon_{\infty}^{\text{out}}(x)=\max_{{\langle} a,b{\rangle}\in \mathcal{C}_{\text{out}}(x) }w(a,b) $$

((15))

$$ \varepsilon_{\infty}^{\text{in}}(x)=\max_{{\langle} a,b{\rangle}\in \mathcal{C}_{\text{in}}(x) }w(a,b) $$

((16))

Let ${\varepsilon }_{\infty \downarrow }^{\text {out}}$ be the minimum value of the energy ${\varepsilon }_{\infty }^{\text {out}}(x)$, that is:

$$ \varepsilon_{\infty\downarrow}^{\text{out}} =\min \{{\varepsilon}_{\infty}^{\text{out}}(x):x\in {\mathcal {X}}(\mathcal{S}_{{o}},\mathcal{S}_{{b}})\} $$

((17))

Similarly, for ${\varepsilon }_{\infty }^{\text {in}}(x)$, we have:

$$ \varepsilon_{\infty\downarrow}^{{in}} =\min \{{\varepsilon}_{\infty}^{{in}}(x):x\in {\mathcal{X}}(\mathcal{S}_{{o}},\mathcal{S}_{{b}})\} $$

((18))

Therefore, we have the following sets of solutions:

$$ \mathcal{X}_{\infty}^{\text{out}}(\mathcal{S}_{{o}},\mathcal{S}_{{b}}) = \left\{ x\in \mathcal{X}(\mathcal {S}_{{o}},\mathcal{S}_{{b}}): {\varepsilon}_{\infty}^{\text{out}}(x) = {\varepsilon}_{\infty\downarrow}^{\text{out}} \right\} $$

((19))

$$ \mathcal{X}_{\infty}^{\text{in}}(\mathcal{S}_{{o}},{\mathcal{S}}_{{b}}) = \left\{x\in {\mathcal{X}}({\mathcal {S}}_{{o}}, \mathcal{S}_{{b}}): {\varepsilon}_{\infty}^{\text{in}}(x) = {\varepsilon}_{\infty\downarrow}^{\text{in}} \right\} $$

((20))

The ORFC algorithms on digraphs have the following definitions based on cut in graph:For the outer cut ‘ out’ with one internal seed s ₁,

((21))

and in the case of multiple internal seeds,

((22))

For the inner cut ‘ in’ with one internal seed s ₁,

((23))

and in the case of multiple internal seeds,

((24))

5.3 ORFC algorithm based on graph cut

In order to show the proposed algorithms, we need the following definition:

Definition 1 (Directed connected component).

For a given vertex x of a digraph G, the directed connected component of basepoint x is the set, denoted by DCC _G (x), of all the successors of x in G (i.e., all the nodes that are reachable from vertex x by some path).

Figure 5 illustrates the steps of Algorithm 1.

Figure 5

Algorithm .(a) Image as a digraph. (b) Initialization of IFT with background seed S _b for computing the connectivity value V _b (s _i ) using the connectivity function $f^{S_{b}}_{\text {min}}$. (c) Result of Step 1: The value V _b (s _i )=1 is computed by the IFT. (d) Step 2: The graph G _≤. (e,f) Step 3: The transpose graph of G _≤ and finally, the object’s pixels from the DCC.

To prove the correctness of the above algorithms, we need the following lemma:

Lemma 1.

For a given weighted digraph G, and sets of seeds S _o and S _b , such that S _o ={s _i }, we have that ${\varepsilon }_{\infty \downarrow }^{{in}} = V_{{b}}(s_{i})$, and ${\varepsilon }_{\infty \downarrow }^{out} = V_{{b}}^{\not \parallel }(s_{i})$.

Proof.

We will prove Lemma 1 for ${\varepsilon }_{\infty \downarrow }^{\text {in}} = V_{{b}}(s_{i})$, but the case ${\varepsilon }_{\infty \downarrow }^{\text {out}} = V_{{b}}^{\not \parallel }(s_{i})$ has an essentially identical proof. The proof is based on the following statement: (1) For the given strongly connected digraph G, if we remove all arcs 〈a,b〉, such that $w(a,b)<{\varepsilon }_{\infty \downarrow }^{\text {in}}$, we then obtain a new digraph G ^′ where there still exists a path from ${\mathcal {S}}_{{b}}$ to s _i (i.e., $\exists \pi _{t \rightsquigarrow s_{i}}$ where $t \in {\mathcal {S}}_{{b}}$).

This statement can be proven by contraction. Let T be the set of pixels reachable from $\mathcal {S}_{{b}}$ in G ^′ (i.e., $T = \bigcup _{x \in {\mathcal {S}}_{{b}}} \text {DCC}_{G'}(x)$). If there is no path from ${\mathcal {S}}_{{b}}$ to s _i in G ^′, then we have that s _i ∉T. Therefore, we have a partition of the vertices into two disjoint sets T and V∖T. Note that its corresponding cutting arcs ${\langle } a,b{\rangle } \in {\mathcal {C}}_{\text {in}}(\chi _{V/T})$ all have $w(a,b)<{\varepsilon }_{\infty \downarrow }^{\text {in}}$ in G. Consequently, ${\mathcal {C}}_{\text {in}}(\chi _{V/T})$ has a better cut value than ${\varepsilon }_{\infty \downarrow }^{\text {in}}$, which is a contradiction by Equation 18.

From statement (1), we may conclude that there is a path from ${\mathcal {S}}_{{b}}$ to s _i in G, which is composed only by arcs ${\langle } a,b{\rangle } \colon w(a,b)\geq {\varepsilon }_{\infty \downarrow }^{\text {in}}$. Hence, the connectivity value V _b (s _i ) of an optimum path from ${\mathcal {S}}_{{b}}$ to s _i (Equation 2) cannot be lower than ${\varepsilon }_{\infty \downarrow }^{\text {in}}$, i.e., $V_{{b}}(s_{i}) \geq {\varepsilon }_{\infty \downarrow }^{\text {in}}$ (2).

Consider the set of cutting arcs ${\mathcal {C}}_{\text {in}}(x_{\text {opt}})$ of an optimum solution $x_{\text {opt}} \in \mathcal {X}_{\infty }^{\text {in}}(\{s_{i}\},{\mathcal {S}}_{{b}})$. By definition (Equations 16 and 18), we have that $w(a,b) \leq {\varepsilon }_{\infty \downarrow }^{\text {in}}$ for all ${\langle } a,b{\rangle } \in {\mathcal {C}}_{\text {in}}(x_{\text {opt}})$. An optimum path $\pi _{S_{{b}} \rightsquigarrow s_{i}}$, from ${\mathcal {S}}_{{b}}$ to s _i , must necessarily pass through some arc of ${\mathcal {C}}_{\text {in}}(x_{\text {opt}})$. So its connectivity value $f^{{\mathcal {S}}_{{b}}}_{\min }(\pi _{S_{{b}} \rightsquigarrow s_{i}}) = V_{{b}}(s_{i})$ cannot be greater than ${\varepsilon }_{\infty \downarrow }^{\text {in}}$, i.e., $V_{{b}}(s_{i}) \leq {\varepsilon }_{\infty \downarrow }^{\text {in}}$ (3).

From the above conditions (2) and (3), we may conclude that the only valid configuration is $V_{{b}}(s_{i}) = {\varepsilon }_{\infty \downarrow }^{\text {in}}$.

For the sake of simplicity, we will only discuss here the proof of correctness of the algorithm, in terms of Equation 23, where s ₁=s _i and ${\varepsilon }_{\infty \downarrow }^{\text {in}}=V_{{b}}(s_{i})$ (Lemma 1). The algorithm for has an essentially identical proof.

First, we need to prove that the characteristic function ${\chi }_{O} of O = \text {DCC}_{G_{\le }^{T}}(s_{i})$ is an optimum solution in $\mathcal {X}_{\infty }^{\text {in}}(\{s_{i}\},{\mathcal {S}}_{{b}})$. Note that, in the digraph $G_{\le }^{T}$, there are no arcs from pixels in $\text {DCC}_{G_{\le }^{T}}(s_{i})$ to pixels in $V \backslash \text {DCC}_{G_{\le }^{T}}(s_{i})$; otherwise, the list of successors of s _i in $G_{\le }^{T}$, given by $\text {DCC}_{G_{\le }^{T}}(s_{i})$, would not be complete. These arcs were removed in Step 2 of the algorithm and, therefore, have no values greater than ${\varepsilon }_{\infty \downarrow }^{\text {in}}$, so the characteristic function of $\text {DCC}_{G_{\le }^{T}}(s_{i})$ must be an optimum solution in $\mathcal {X}_{\infty }^{\text {in}}(\{s_{i}\},{\mathcal {S}}_{{b}})$.

The other conditions in Equation 23 force to constitute the smallest object in ${\mathcal {X}}_{\infty }^{\text {in}}(\{s_{i}\},{\mathcal {S}}_{{b}})$. Note that any object composed by a set of pixels T, such that there are arcs from pixels in T to pixels in V∖T in the digraph $G_{\le }^{T}$, cannot be an optimum solution in ${\mathcal {X}}_{\infty }^{\text {in}}(\{s_{i}\},{\mathcal {S}}_{{b}})$ because these arcs have corresponding anti-parallel arcs in G _≤, pointing toward object pixels, with values greater than ${\varepsilon }_{\infty \downarrow }^{\text {in}}$, leading to a worse inner cut. Since all proper subsets of $\text {DCC}_{G_{\le }^{T}}(s_{i})$ still have some outgoing arcs in the digraph $G_{\le }^{T}$ and, consequently, incoming arcs in G _≤, we have that they are not optimum. Therefore, $\text {DCC}_{G_{\le }^{T}}(s_{i})$ is the smallest optimum solution.

To solve the case of with multiple internal seeds, according to Equation 24, we need to repeat the execution of Algorithm 1 for each internal seed. However, the following proposition applies in the case of :

Proposition 1.

For a given digraph G=〈V,E,w〉 and seed $s_{i} \in {\mathcal {S}}_{{o}}$, consider the residual digraph $G_{s_{i}} = G_{\le } = {\langle } V,\{{\langle } a,b{\rangle } \in E \colon w(a,b) > V_{{b}}(s_{i}) \},w\rangle$ (step 2 of Algorithm 1). For any arbitrary seeds $s_{1} \in {\mathcal {S}}_{{o}}$ and $s_{2} \in {\mathcal {S}}_{{o}}$, if V _b (s ₁)≤V _b (s ₂) and $s_{2} \in DCC_{G_{s_{1}}^{T}}(s_{1})$, then we have that $DCC_{G_{s_{2}}^{T}}(s_{2}) \subset {DCC}_{G_{s_{1}}^{T}}(s_{1})$.

If we sort the seeds s _i in S _o according to their V _b (s _i ) values, then we can process the seeds in increasing order of values, allowing us to avoid the reprocessing of pixels by skipping the seeds that were already assigned to the object, greatly improving the running time.

One important thing to note is that ORFC encompasses RFC as a particular case whenever the parameter α is set to zero.

6 Hybrid segmentation: ORFC and graph cut

In this section, we follow the same key ideas from [29], which proposes a hybrid approach combining the strengths of relative fuzzy connectedness and min-cut/max-flow algorithm.

The GC natively supports the soft constraint of boundary polarity and will be denoted as oriented graph cut (OGC). It ($A_{\text {OGC}}^{\text {out}}(S_{{o}},S_{{b}})$) solves the ε ₁-minimization problem by considering the arcs that limit the flow from the source to the sink and consequently minimizes the sum of the arcs pointing from object to background pixels (i.e., the outer cut) [11]. The minimization of the sum of the arcs of the inner cut ($A_{\textit {OGC}}^{\text {in}}(S_{{o}},S_{{b}})$) can be obtained by inverting the source and sink nodes or by reversing all arcs by computing GC over the graph’s transpose G ^T.

Basically, by considering in [29] a directed weighted graph, with ORFC in place of RFC, we have the ORFC+GC hybrid approach (Figure 6) as follows:

Figure 6

An example result of ORFC+GC. The segmentation of the wrist by: (a) IRFC [9], (b) OIFT [22], (c) RFC+GC [29], and (d) ORFC+GC.

Figure 7 illustrates the steps of Algorithm 4.

Figure 7

Algorithm .(a) Input image with seeds S _o and S _b . (b) . (c) . (d) $A_{\text {OGC}}^{\text {out}}(P, Q)$.

7 Experimental results

In the first experiment, we used 40 slice images from real MR images of the foot. We performed the segmentation of the bones calcaneus and talus for all the methods (IRFC [9], RFC [26], OIFT [23], RFC+GC [29], OGC [11] - the graph cut with boundary polarity, ORFC, and the proposed hybrid method ORFC+GC), for different seed sets automatically obtained by eroding and dilating the ground truth at different radius values. By varying the radius value, we can repeat the segmentation for different seed sets and trace accuracy curves using the dice coefficient of similarity and error curves of false positive (normalized by the object size). However, in order to generate a more challenging situation, we considered a larger radius of dilation for the external seeds (twice the value of the inner radius), resulting in an asymmetrical arrangement of seeds. In the second experiment, 40 slice images from CT thoracic studies of 10 subjects were used to segment the liver following the same procedure for seed selection (Figure 8). In all cases, the ground truth data was obtained from an expert of the Radiology Department at the University of Pennsylvania.

Figure 8

Example result by OIFT and ORFC.(a) A CT slice image of the liver with seeds obtained by eroding and dilating the true segmentation. (b,c) The segmentation results with a postprocessing by closing of holes for: (b) OIFT and (c) ORFC.

Several different procedures can be adopted for δ(a,b) [34,35]. For example, Figures 3 and 9 show some results for user-selected markers using the image-based weight assignment from [36]. For the sake of simplicity, in the quantitative experiments, we adopted the weight assignment δ(a,b)=K−|G(a)+G(b)|, where G(a) denotes the gradient magnitude of the Sobel operator. In Equation 8, α could be in the range of [0,1], we considered α=0.5 in all experiments involving OIFT, OGC, ORFC, and ORFC+GC and α=0.0 in the case of undirected approaches. The value α=0.5 is the default value adopted in experimental results [22,23], which is a more well balanced configuration. For low values (α≈0.0), the oriented methods (e.g., ORFC) degenerate into their counterpart undirected approaches (e.g., RFC), and for high values, the oriented methods may become more sensitive to noise. We used for the foot bones, since they present transitions from dark to bright pixels and for the liver, since it has the opposite orientation.

Figure 9

Example results for the different methods.(a) A talus bone in an MRI slice of a foot with user-selected markers. (b) The image-based arc weight assignment from [36]. (c-f) The segmentation results for: (c) IRFC, (d) RFC, (e) OIFT, and (f) ORFC. The segmentations are shown highlighted with a yellow overlay.

Figure 10 shows the accuracy and error curves, and Figure 11 shows the running time curves for all the methods. Since the considered objects are not expected to have holes, we considered a post-processing by closing of holes (CoH) [37] for RFC and ORFC. Indeed, RFC is known to potentially exclude regions inside the object that are surrounded by strong edges. Note that its oriented counterpart method, ORFC, also inherits this property.

Figure 10

The experimental curves for the 2D datasets. The mean accuracy curves (dice coefficient of similarity) and the normalized false positive curves, using non-equally eroded-dilated seeds, for segmenting: (a,b) calcaneus, (c,d) talus, and (e,f) liver.

Figure 11

The running time curves for the 2D datasets. The computational time using non-equally eroded-dilated seeds, for segmenting: (a) calcaneus, (b) talus, and (c) liver.

In general, the results show that ORFC can achieve higher accuracy values than RFC, with low false positive errors. Hence, it could be combined with other methods in powerful hybrid approaches. Indeed, the hybrid approach ORFC+GC showed the best results for the calcaneus bone, being more robust than the graph cut with respect to the seed choice. OGC presents a drop of accuracy for higher radius values due to the known ‘shrinking problem’ of graph cut. ORFC presented the best results for the liver segmentation, taking advantage of its homogeneous interior. For the talus, ORFC showed a similar accuracy than ORFC+GC, but with a lower false positive rate and being less time-consuming.

We also repeated the experiments using two three-dimensional datasets. In the former case, a MRI-T1 dataset of the human brain was used to segment the cerebellum. The images were taken from 20 normal subjects from both genders, in the age range from 16 to 49 years. The images were acquired with a 2T Elscint scanner and at a voxel size of 0.98×0.98×1.00mm³. The cerebellum is connected to the rest of the brain through the brain stem and through its top due to partial volume. The absence of a clear boundary between these structures poses a significant challenge for segmentation. For the second dataset, we considered a skull stripping task (Figure 12) (i.e., to eliminate background, bones, eyes, skin, and blood vessels) using ten 3 Tesla MRI-T1 images, that include the head and, at least, a small portion of the neck of male and female adults with normal brains.

Figure 12

Example of skull stripping.(a) The ground truth. The segmentation results for: (b) OGC, (c) RFC, (d) ORFC, (e) IRFC, (f) OIFT, (g) RFC+GC, and (h) ORFC+GC.

Figures 13 and 14 show the experimental curves. The R F C and O R F C methods performed poorly on these datasets due to the lack of a clear contrast between the structures. Nevertheless, the hybrid approach ORFC+GC showed the overall best results, demonstrating the importance of hybrid methods, and making clear that, even in these cases, O R F C can help to improve the graph cut delineation and to reduce its running time.

Figure 13

The experimental curves for the 3D datasets. The mean accuracy (dice coefficient) and normalized false positive curves, using non-equally eroded-dilated seeds, for segmenting: (a,b) cerebellum dataset and (c,d) skull stripping dataset.

Figure 14

The running time curves for the 3D datasets. The computational time using non-equally eroded-dilated seeds, for segmenting: (a) cerebellum dataset and (b) skull stripping dataset.

8 Conclusions

In this paper, we introduced the ORFC technique and showed that it can effectively exploit the boundary polarity improving the results in relation to its predecessor RFC. We also presented a powerful hybrid approach, which outperforms the previous works [29,30]. As future work, we plan to investigate the theoretical relations between ORFC and OIFT, the usage of shape constraints in the ORFC (similar to what was done in [25]), and to combine the proposed methods with fuzzy object models [38-41] in order to get a fully automatic segmentation result.