Skip to main content

Advertisement

You are viewing the new article page. Let us know what you think. Return to old version

Graph matching using conformal module

Abstract

Graph matching and classification play fundamental roles in computer vision. The computational complexity of the conventional method based on a spectrum method is high, which prevents it from handling large graphs in practice. This work proposes a novel framework for tackling the challenge by using conformal module. We apply the classical Hodge theory from differential manifold to the graph setting and compute the combinatorial conformal invariant of the graph, called as conformal module, which can be used as the fingerprint for the graph. The method is applicable for viewpoint classification and posture detection. The experimental results demonstrate the efficiency and efficacy of the proposed method.

Introduction

Graph matching has been widely used to solve a large variety of problems in computer vision. Fundamentally, graph matching aims at finding correspondences between two sets of features extracted from images. When matching or recognizing structural objects, graph matching does well on considering the pairwise node interactions [1]. Furthermore, when there are correspondences between two images, graph matching can finish many vision tasks such as object categorization [2], feature tracking [3], symmetry analysis [4], and posture recognition [5, 6].

In the past thirty years, extensive research has been carried out on graphing matching [7], however, there still exists several challenges in current methods. For example, almost all matching algorithms require high time complexity, so it is difficult to deal with large graphs. No methods have been provided to deal with graphs in 3D vision field, which means that graphs embedded in high genus surfaces cannot be matched using current approaches. Many techniques are proposed to solve graph matching in terms of rigid transformation, which can hardly handle the circumstance where two graphs to be matched have large non-rigid transformations. Although the third issue is considered in [8], the other two issues are still open. Our main contributions conclude as follows: (1) the proposed method can discover the combinatorial invariant of a given graph, (2) the proposed method can tolerate non-rigid transformation on a given graph, (3) the proposed method can solve the matching problem in linear time, (4) the proposed method can be applied on not only planar graphs, but also those embedded in high genus surfaces.

The main mathematical theory in our method is based on combinatorial Hodge theory. There is some work on surface registration [9, 10] and shape analysis [11] by the combinatorial Hodge theory. In the textbook [12], Gu et al. introduced the detailed theoretic treatment on combinatorial Hodge theory. These works focus on computing the conformal invariants and conformal mappings induced by the Riemannian structure of the input surfaces. By contrast, current work focuses on computing the conformal modules induced solely by the combinatorial structure of the graphs. The process is that we apply combinatorial Hodge theory to compute the combinatorial harmonic functions on graphs. The solution of harmonic functions is equivalent to solving combinatorial Laplace matrices. The combinatorial harmonic 1-forms can give the combinatorial conformal invariant of a given graph, called as conformal module, which can be used as the fingerprint of a given graph. So the conformal module can be used to deal with some matching problems. In this paper, we compute the conformal modules by the combinatorial structure of the given graphs for viewpoint classification and posture detection.

Previous works

Graph matching problem has been an active field in computer science and mathematics. There is a large literature for this topic in the past thirty years, so a thorough survey is beyond the scope of the current work. In the following, we only review the most related work. For more detailed treatment, we refer readers to [7]. There usually exists two categories of graph matching methods: exact matching and inexact matching.

Exact matching aims at graph isomorphism, where the interest is in an exact matching between nodes and edges. Most of approaches to find the graph isomorphism are based on Tree Search. Due to the brute Tree Search algorithm, some improvements are developed. Ullmann proposes one of the most important and popular tree search algorithms [13]. He prunes the unfruitful matches which are not consistent with current partial matching. Cordella suggests another interesting approach based on Tree Search [14, 15]. He provides a heuristic based on analysis of the sets of nodes adjacent to the ones already considered in current partial matching. Except for Tree Search, some other techniques are proposed. According to group theory, an approach was proposed by McKay [16]. He defines equivalence classes for graphs in terms of isomorphism and does the equality verification in O(n2). There are other methods based on machine learning techniques to speed up the matching against a large library of graphs [17, 18]. But they are used for filtering out the unmatched candidates not for matching.

Inexact matching focuses on mapping between graphs with weighted attributes on nodes and edges. The solution is a quadratic assignment problem and the optimization is NP-hard [19]. Looking for better optimization strategies is important for major research in graph matching. According to the relaxations of the constraints on a one-to-one mapping F between the nodes in both graphs [8], methods can be generally classified into several categories. The first is to approximate the constraints as an orthogonal one FTF = I, which is called as spectral method. Under the orthogonal constraint, optimizing graph matching can be solved in closed form as an eigen value problem [20,21,22]. Furthermore, Leordeanu and Hebert relax F to be of unit length \( {\left\Vert F\right\Vert}_2^2=1 \) for handling more complex problems in computer vision [23]. The work in [24, 25] proposes a spectral method to decompose the graph into sub-graphs, where the process can be cast into a hierarchical framework and suitable for parallel computation. The second group of methods relaxes F to be a doubly stochastic matrix, the convex hull of F. Under this constraint, optimizing graph matching can be solved as a non-convex quadratic programming problem and a local optimization could be done within several proposed strategies. For instance, Gold and Rangarajan propose the graduated assignment algorithm for iteratively solving a series of linear approximations of the cost function in the form of Taylor expansion. The work of [26] proposes a path-following algorithm. Besides the optimization-based work, probabilistic frameworks are also useful for interpreting and solving graph matching [27]. Several work aims at higher-order tensor factorization [8, 28, 29], where higher order geometrical relations make graph matching invariant to rigid transformation. Due to a small increment in the order of relations induces a combination explosion of data, it is only suitable for very sparse graphs. Furthermore, this method cannot deal with non-rigid transformations.

Method

This section briefly introduces the theoretical background for combinatorial conformal module. We apply algebraic topology method to consider our matching problem. Algebraic topology focuses on studying topological problems by algebraic methods and uses simple linear algebraic methods to solve these problems. The detailed description of algebraic topology methodology can be seen in [12], where the authors consider these methods on surfaces with Riemann structure. In this paper, we apply algebraic topology method to the graph setting and consider combinatorial Hodge theory on graphs. Combinatorial Hodge theory can deduce a unique combinatorial conformal module for each graph, so it can be viewed as the fingerprint of a given graph.

We consider all objects on a three-connected planar graph G which is embedded on a topological surface S. A planar graph means that there are no edge crossings on G and different edges of the graph can only intersect at the endpoints. So each face of G can be viewed as a topological disk. We usually use a triple G = (V, E, F) to represent a graph, where V is the vertex set, E is the edge set and F is the face set. At the first step, we need to introduce some tools about computational topology. These similar concepts are defined on surfaces with Riemann structure in [12]. Here we use them on graphs.

The vertex set is denoted as

$$ V=\left\{{v}_1,{v}_2,\cdots, {v}_n\right\}. $$
(1)

Each oriented edge e is denoted as

$$ e=\left[{v}_i,{v}_j\right]\kern0.24em \mathrm{or}\kern0.24em e=-\left[{v}_j,{v}_i\right], $$
(2)

where vi, vj are its two endpoints.

A face f is represented as

$$ f=\left[{v}_{i_1},{v}_{i_2},\cdots, {v}_{i_k}\right], $$
(3)

where \( {v}_{i_k} \) is its vertex and (i1, i2, , ik) is any cyclic permutation of (1, 2, , k).

We call the vertices of G as 0-dimensional cells, the oriented edges of G as 1-dimensional cells and the faces of G as 2-dimensional cells. Their linear combinations can form a linear chain space. We will give some definitions on the chain space of a graph.

The k-dimensional chain group of a graph G = (V, E, F) is given by

$$ {C}_k\left(G,\mathrm{\mathbb{Z}}\right)=\left\{\sum \limits_{i=1}^{n_k}{\xi}_i{\varepsilon}_k^i,{\xi}_i\in \mathrm{\mathbb{Z}}\right\}, $$
(4)

where \( {\varepsilon}_k^i \) denotes the ith k-dimensional cell of G. We call each element of Ck(G, Z) as a k-chain of G.

Definition 1: The k-dimensional boundary operator k : Ck(G, ) → Ck − 1(G, ) on graph G is defined as

$$ {\partial}_k\left(\sum \limits_{i=1}^{n_k}{\xi}_i{\varepsilon}_k^i\right):= \sum \limits_{i=1}^{n_k}{\xi}_i{\partial}_k\left({\varepsilon}_k^i\right). $$
(5)

The boundary operator has the linear property. In this paper, we only consider the low dimensional boundary operator, for instance,

$$ {\partial}_0{v}_i=0,{\partial}_1\left[{v}_i,{v}_j\right]={v}_j-{v}_i,{\partial}_2\left[{v}_i,{v}_j,{v}_k\right]=\left[{v}_i,{v}_j\right]+\left[{v}_j,{v}_k\right]+\left[{v}_k,{v}_j\right]. $$
(6)

The k-chain in the kernel of k satisfies

$$ \mathit{\operatorname{Ker}}\ {\partial}_k=\left\{{\varepsilon}_k\in {C}_k\left(G,\mathrm{\mathbb{Z}}\right)|{\partial}_k{\varepsilon}_k=0\right\}. $$
(7)

We call each element in Ker ∂k as a closed chain. The closed chains are closed loops and the boundaries of any 2-dimensional patches are called as boundary loops.

The k-chain in the image of k + 1 satisfies

$$ Img\ {\partial}_{k+1}=\left\{{\varepsilon}_k\in {C}_k\left(G,Z\right)|\exists {\varepsilon}_{k+1}\in {C}_{k+1}\left(G,Z\right),{\varepsilon}_k={\partial}_{k+1}{\varepsilon}_{k+1}\right\}. $$
(8)

Each element of Img ∂k + 1 is called as an exact chain.

Property 1: The boundary of a boundary is empty:

$$ {\partial}_k\circ {\partial}_{k+1}\equiv 0. $$
(9)

The proof can be seen in [12].

From the above property, we can easily obtain that Img ∂k + 1 is a sub-linear space of Ker ∂k. The complementary space is called as the homology group of the graph.

Definition 2: Suppose k : Ck(G, ) → Ck − 1(G, ) is the k-dimensional boundary operator on a graph G. Ker ∂k = {εkCk(G, )| kεk = 0} is the kernel of k. Img ∂k + 1 = {εkCk| εk + 1Ck + 1, εk = k + 1εk + 1} is the image of k + 1. The k-dimensional homology group of the graph G is defined as the quotient group, satisfying

$$ {H}_k\left(G,\mathrm{\mathbb{Z}}\right):= \frac{\mathit{\operatorname{Ker}}\ {\partial}_k}{Img\ {\partial}_{k+1}}. $$
(10)

Each element in Hk(G, Z) is called a k-dimensional homologous class. A closed k-chain εk  ker ∂k represents a homologous class, denoted as [εk] Hk(G, Z). Each element in the homologous class can form the basis of homology group. The similar algorithm in [12] is used to compute the homology basis of a discrete surface. Figures 1 and 2 show the homology basis of two graphs with different topology.

Fig. 1
figure1

Homology group basis for a graph embedded on a genus one surface

Fig. 2
figure2

Homology group basis for a graph embedded on a genus two surface

In order to do computation on graphs, we define the cohomology group which is the dual of the homology group. In fact, cohomology group is the space of linear functions defined on homology group. The following discussions are considered in parallel because all the objects are dual of the above counterparts such as co-chain and exterior differential operator.

The k-dimensional co-chain space of the graph G is a linear function defined on k-dimensional chain space and given by

$$ {C}^k\left(G,\mathrm{\mathbb{R}}\right)=\left\{{\omega}^k:{C}_k\left(G,Z\right)\to \mathrm{\mathbb{R}}\right\}. $$
(11)

We call each element in Ck(G, R) as a k-form.

The dual of the boundary operator is called as the exterior differential operator d.

Definition 3: The exterior differential operator dk : Ck(G, ) → Ck + 1(G, ) is defined as follows:

$$ {d}_k{\omega}^k\left({\varepsilon}_{k+1}\right):= {\omega}^k\left({\partial}_{k+1}{\varepsilon}_{k+1}\right), $$
(12)

where ωkCk(G, R), εk + 1Ck + 1(G, Z).

Corresponding to closed chain, we have that any k-form in the kernel of dk satisfies

$$ \mathit{\operatorname{Ker}}\ {d}_k=\left\{{\omega}^k\in {C}^k\left(G,\mathrm{\mathbb{R}}\right)|{d}_k{\omega}^k=0\right\}. $$
(13)

We call each element in Ker dk as a closed k-form.

Similarly, any k-form in the image of dk satisfies

$$ Img\ {d}_{k-1}=\left\{{\omega}^k\in {C}^k\left(G,R\right)|\exists {\omega}^{k-1}\in {C}^{k-1}\left(G,R\right),{\omega}^k={d}_{k-1}{\omega}^{k-1}\right\}. $$
(14)

We call each element in Ker dk as an exact k-form.

From Property 1, we obviously obtain the following property.

Property 2: The exterior differential operator dk : Ck(G, ) → Ck + 1(G, ) is a linear operator with the property

$$ {d}_k\circ {d}_{k-1}\equiv 0. $$
(15)

According to property 2, we have that exact forms are closed. Therefore, Img dk − 1 is a sub-linear space of Ker dk. The complementary space is defined as the cohomology group of the graph.

Definition 4: Suppose the linear operator dk : Ck(G, ) → Ck + 1(G, ) is the exterior differential operator on a graph G. Ker dk = {ωkCk(G, )| dkωk = 0} is the kernel of dk. Img dk − 1 = {ωkCk(G, R)| ωk − 1Ck − 1(G, R), ωk = dk − 1ωk − 1} is the image of dk − 1. The k-dimensional cohomology group of the graph G is defined as the quotient group, satisfying

$$ {H}^k\left(G,R\right):= \frac{\mathit{\operatorname{Ker}}\ {d}_k}{Img\ {d}_{k-1}}. $$
(16)

A graph G embedded on a topological surface S can induce a CW-cell decomposition. The Poincare dual of G is denoted as \( \overline{G} \) which is also a graph embedded on S such that each vertex vi of G corresponds to a face \( {\overline{v}}_i \) of \( \overline{G} \) and each face fj of G corresponds to a vertex \( {\overline{f}}_j \) of \( \overline{G} \). Furthermore, each oriented edge e of G corresponds to an oriented edge \( \overline{e} \) on \( \overline{G} \) as follows: if e = [vi, vj], \( \overline{e}={\overline{v}}_i\cap {\overline{v}}_j \). Furthermore, if the left face of e is fl and the right face of e is fr, \( \overline{e}=\left[{\overline{f}}_r,{\overline{f}}_l\right] \).

Definition 5: Suppose a graph and its dual are denoted as G and \( \overline{G} \) respectively. The Hodge star operator \( {}^{\ast }:{C}^k\left(G,\mathrm{\mathbb{R}}\right)\to {C}^{2-k}\left(\overline{G},\mathrm{\mathbb{R}}\right) \) is a linear operator, satisfying

$$ {}^{\ast}\omega \left(\overline{\varepsilon}\right):= \omega \left(\varepsilon \right), $$
(17)

where ωCk(G, R), εCk(G, Z), \( \overline{\varepsilon}\in {C}_{2-k}\left(\overline{G},Z\right) \), and \( {}^{\ast}\omega \in {C}^{2-k}\left(\overline{G},R\right) \).

Combining definition 3 and definition 5, we can obtain another linear operator σk : Ck(G, R) → Ck − 1(G, R), satisfying

$$ {\sigma}_k{=}^{\ast }{d_k}^{\ast }. $$
(18)

The operator σk can be viewed as the adjoint of the exterior differential operator, called as the co-differential operator. Combining operator dk and σk, we can define the Laplace operator.

Definition 6: Suppose dk : Ck(G, ) → Ck + 1(G, ) is the exterior differential operator on a graph \( G.{}^{\ast }:{C}^k\left(G,\mathrm{\mathbb{R}}\right)\to {C}^{2-k}\left(\overline{G},\mathrm{\mathbb{R}}\right) \) is the Hodge star operator on G. σk = dx : Ck(G, R) → Ck − 1(G, R) is the co-differential operator on G. The Laplace operator Δ : Ck(G, R) → Ck(G, R) is defined as

$$ \varDelta := {\sigma}_{k+1}{d}_k+{d}_{k-1}{\sigma}_k. $$
(19)

Obviously, the Laplace operator is symmetric, that is, for any ξ, ηCk(G, R),

$$ \left(\varDelta \zeta, \eta \right)=\left(\zeta, \varDelta \eta \right). $$
(20)

Furthermore,

$$ \left(\varDelta \zeta, \zeta \right)\ge 0. $$
(21)

We mainly consider functions whose values are equal to zero with the action of the Laplace operator. These functions have inspiring mathematical and physical properties. In this paper, we will compute these functions on the graph setting. Based on the combinatorial structure of a graph, the solution of harmonic functions is equivalent to solving combinatorial Laplace matrices on the graph. Finally, these combinatorial harmonic functions can give out the combinatorial conformal invariant of a given graph, called as conformal module, which can be used as the fingerprint of a given graph. So, the conformal module can be applied into matching applications on graphs.

Definition 7: Suppose dk : Ck(G, ) → Ck + 1(G, ) is the exterior differential operator on a graph G. σk = dx : Ck(G, R) → Ck − 1(G, R) is the co-differential operator on G. Δ = σk + 1dk + dk − 1σk : Ck(G, R) → Ck(G, R) is the Laplace operator on G. A k-form ωCk(G, ) is called to be harmonic, if

$$ \varDelta \omega =0. $$
(22)

Since (Δω, ω) = |dkω|2 + |σkω|2 = 0, we can easily have the following property.

Property 3: A harmonic k-form ωCk(G, ) satisfies

$$ \left\{\begin{array}{ccc}{d}_k\omega & =& 0\\ {}{\sigma}_k\omega & =& 0.\end{array}\right. $$
(23)

From the physical viewpoint, the exterior differential operator dk is the curl operator and the co-differential operator σk is the divergence operator. So a harmonic k-form means that it is both curl-free and divergence-free.

The group of all the harmonic k-forms is denoted as \( {H}_{\Delta}^k\left(G,\mathrm{\mathbb{R}}\right) \), namely

$$ {H}_{\Delta}^k\left(G,\mathrm{\mathbb{R}}\right)=\mathit{\operatorname{Ker}}\ {d}_k\cap \mathit{\operatorname{Ker}}\ {\sigma}_k. $$
(24)

According to Hodge theory, any a k-form can be decomposed into an exact form, a co-exact form and a harmonic form.

Theorem 1: Suppose ω is a k-form. dk − 1 : Ck − 1(G, ) → Ck(G, ) is the exterior differential operator on a graph G. σk = dx : Ck(G, R) → Ck − 1(G, R) is the co-differential operator on G. Then there is a unique (k − 1)-form α, (k + 1)-form β and a harmonic k-form γ, such that

$$ \omega ={d}_{k-1}\alpha +{\sigma}_{k+1}\beta +\gamma; $$
(25)

moreover, such kind of decomposition is unique.

The detailed proof can be checked in [12].

For example k = 2, suppose G is embedded on a closed surface with genus g.{γ1, γ2, , γ2g} is the generator of 1-dimensional homology group H1(G, Z). Let ω1, ω2 be two harmonic 1-forms with respect to the homology equivalence class, namely

$$ {\int}_{\gamma_k}{\omega}_1={\int}_{\gamma_k}{\omega}_2,k=1,2,\cdots, 2g. $$
(26)

Due to the linear property, we have

$$ {\displaystyle \begin{array}{ccccc}d\left({\omega}_1-{\omega}_2\right)& =& d{\omega}_1-d{\omega}_2& =& 0,\\ {}\sigma \left({\omega}_1-{\omega}_2\right)& =& {\sigma \omega}_1-{\sigma \omega}_2& =& 0.\end{array}} $$
(27)

So ω1 − ω2 is harmonic and \( {\int}_{\gamma_k}\left({\omega}_1-{\omega}_2\right)={\int}_{\gamma_k}{\omega}_1-{\int}_{\gamma_k}{\omega}_2=0.\left({\omega}_1-{\omega}_2\right) \) is exact. There is a function f : G → R such that ω1 − ω2 = df. According to the maximum principle and the empty boundary, f has no extremal value and is constant. That means the uniqueness holds: ω1 − ω2 = df = 0.

Suppose ωH1(G, R) is closed and f : G → R is a smooth function. So ω + df and ω lie in the same cohomology class. The solution of σ(ω + df) = 0 is equivalent to the solution of Poisson equation:

$$ \varDelta f=-\sigma \omega . $$
(28)

In order to compute harmonic functions on a graph, we need define the combinatorial Laplace matrix.

Definition 8: Suppose G = {v1, v2, , vn} is a planar graph. The combinatorial Laplace matrix Δ = (δij) on G is defined as

(29)

where vi~vj means two vertices are adjacent and d(vi) is the degree of vi in the graph. According to theorem 1, the following Corollary is straightforward.

Corollary: The group of all harmonic k-forms on G is isomorphic to the k-dimensional cohomology group, that is

$$ {H}_{\Delta}^k\left(G,\mathrm{\mathbb{R}}\right)\cong {H}^k\left(G,\mathrm{\mathbb{R}}\right). $$
(30)

Suppose ω1 and ω2 are cohomological k-forms, then there is a (k − 1)-form η, such that ω1 = ω2 + . Assume the Hodge decomposition of ω1 is

$$ {\omega}_1=d{\alpha}_1+{\delta \beta}_1+{\gamma}_1=d{\alpha}_1+{\gamma}_1. $$
(31)

1 = 0, dδβ1 = 0, so δβ1 = 0. Directly, we get

$$ {\omega}_2=d\left({\alpha}_1-\eta \right)+{\gamma}_1. $$
(32)

That means two cohomological forms share the same harmonic form. Therefore, each cohomological class in Hk(G, R) corresponds to a unique harmonic form in \( {H}_{\Delta}^k\left(G,R\right) \), which establishes the isomorphism.

Combinatorial Conformal Invariants: Suppose a graph G is embedded on a genus g surface S. ω is a combinatorial harmonic 1-form on G and γ is a closed loop. We can choose a tubular neighborhood of γ,

$$ N\left(\gamma \right):= \left\{\alpha \in G|\alpha \cap \gamma \ne \varnothing \right\}. $$
(33)

We can periodically embed N(γ) to the plane. Each period is represented as a complex number, denoted as ∫γω.

We can choose a canonical homology group basis in H1(G, Z) as {a1, b1, a2, b2, , ag, bg}. Namely, they satisfy the following conditions

$$ {a}_i\cdot {b}_i=1,{a}_i\cdot {a}_j=0,{b}_i\cdot {b}_j=0, $$
(34)

where ab represents the algebraic intersection number. Suppose a closed differential 1-form is ω. The integral of ω on ai

$$ {\int}_{a_1}\omega, {\int}_{a_2}\omega, \cdots, {\int}_{a_g}\omega $$
(35)

is called as the a-period of ω. Similarly,

$$ {\int}_{b_1}\omega, {\int}_{b_2}\omega, \cdots, {\int}_{b_g}\omega $$
(36)

is called the b-period of ω.

For all ai, slicing the surface S to get Si, the boundary of Si is \( \partial {S}_i={a}_i^{+}-{a}_i^{-} \). We construct a function fi such that

$$ {f}_i\left({v}_k\right)=\left\{\begin{array}{c}1,{v}_k\in {a}_i^{+}\\ {}0,{v}_k\in {a}_i^{-}\\ {}\mathrm{random},\mathrm{otherwise}\end{array}\right., $$
(37)

and denote αi = dfi.

For all bj, slicing the surface S to get Sj, the boundary of Sj is \( \partial {S}_j={b}_j^{+}-{b}_j^{-} \). We construct a function gj such that

$$ {g}_j\left({v}_k\right)=\left\{\begin{array}{c}1,{v}_k\in {b}_j^{+}\\ {}0,{v}_k\in {b}_j^{-}\\ {}\mathrm{random},\mathrm{otherwise}\end{array}\right., $$
(38)

and denote βj = dgj.

The combinatorial period matrix of G is as following:

$$ Mod(G)={\left(\underset{a_i}{\int }{\alpha}_j\right)}^{-1}\left(\underset{b_k}{\int }{\beta}_l\right). $$
(39)

For genus one case, the graph G can be embedded into the plane as a parallelogram. The combinatorial period matrix Mod(G) represents the shape of the parallelogram. For high genus graph G, the combinatorial period matrix is the generalization. The combinatorial period matrix can be used as the fingerprint of the embedded graph G.

One face f0 on the graph G is selected as the unique exterior face. Four vertices on the boundary of f0 are chosen, denoted as {v0, v1, v2, v3}, sorted counter clockwisely. The graph G with four corner vertices is called as a topological quadrilateral. The corner vertices divide the boundary of the exterior face into four segments, denoted as left, top, right, and bottom segments, such that v0 is the left lower corner. Then we can compute a harmonic function u : G →  with Dirichlet boundary condition, namely

$$ \left\{\begin{array}{ccc}u\left({v}_i\right)& =& 1,{v}_i\in \mathrm{top},\\ {}u\left({v}_i\right)& =& 0,{v}_i\in \mathrm{bottom},\\ {}\Delta u\left({v}_i\right)& =& 0,\mathrm{otherwise}.\end{array}\right. $$
(40)

In fact, the harmonic 1-form ω is the exterior differential of u, ω = du.

Theorem 2: Suppose G is a topological quadrilateral which is a planar graph with an exterior face f0 and four corner vertices {v0, v1, v2, v3}. The harmonic 1-form maps the topological quadrilateral to a rectangle with width w and height h, then the conformal module of the topological quadrilateral is

$$ Mod\left(G,\left\{{v}_0,{v}_1,{v}_2,{v}_3\right\}\right):= \frac{h}{w}. $$
(41)

This theorem can be proven by an extremal length theory. More detailed proof is shown in [30].

So, there is a conformal mapping ϕ which maps a topological quadrilateral into a canonical rectangle and maps four vertices {v0, v1, v2, v3} into four corner vertices of the rectangle. The ratio of width and height is the conformal module. Four vertices {v0, v1, v2, v3} divide the boundary of the topological quadrilateral into four segments, denoted as Γ0, Γ1, Γ2, Γ3 and Γk = vk + 1 − vk. Solving two Dirichlet equations, there are two harmonic functions f1, f2 : G → R which satisfy the Dirichlet boundary and the Neumann boundary:

$$ \left\{\begin{array}{ccc}\varDelta {f}_1& =& 0,\\ {}{\left.{f}_1\right|}_{\Gamma_0}& =& 0,\\ {}{\left.{f}_1\right|}_{\Gamma_2}& =& 1,\\ {}{\left.\frac{\partial {f}_1}{\partial n}\right|}_{\Gamma_1\cup {\Gamma}_3}& =& 0;\end{array}\right. $$
(42)
$$ \left\{\begin{array}{ccc}\varDelta {f}_2& =& 0,\\ {}{\left.{f}_2\right|}_{\Gamma_3}& =& 0,\\ {}{\left.{f}_2\right|}_{\Gamma_1}& =& 1,\\ {}{\left.\frac{\partial {f}_2}{\partial n}\right|}_{\Gamma_0\cup {\Gamma}_2}& =& 0.\end{array}\right. $$
(43)

where n is the normal vector. The gradient fields f1 and f2 are mutually vertical lines but are not conjugate to each other. A constant λ is necessary for the conformal mapping ϕ with \( \phi =\lambda {f}_2+\sqrt{-1}{f}_1 \). So λ is the width of the canonical rectangle. Due to the harmonic function with conformal invariant property, there is a mapping energy

$$ E\left({f}_1\right)={\int}_G{\left|\nabla {f}_1\right|}^2 dA=\lambda, $$
(44)

and

$$ \phi =E\left({f}_1\right){f}_2+\sqrt{-1}{f}_1. $$
(45)

Experimental results and discussions

All experiments are conducted on a Windows 7 platform, with a single 2.3-GHz Intel CPU, 16-GB RAM memory. All algorithms have been developed in generic C++, compiled using Visual Studio 2010. The sparse linear systems are solved using the numerical library Eigen [31].

The experimental work is to apply the combinatorial Hodge theory to compute the harmonic functions on the graph setting and use these harmonic functions to get the combinatorial conformal module of a given graph. The combinatorial conformal module can be viewed as the fingerprint of a graph. For each image, we make a feature graph for it. Based on the combinatorial conformal modules of graphs, matching work can be done by finding correspondences between two sets of features extracted from images. We test the proposed method in the following applications.

Viewpoint detection

Figure 3 illustrates the experiment for viewpoint detection by the combinatorial conformal modules of graphs. We use the house images with features from CMU house datasets. Although all the images are the same building with the same feature, the combinatorial structure of the Delaunay triangulation varies according to the viewpoint change. If the viewpoints are close to each other, the corresponding combinatorial structure of the feature graph is similar. We expect that the combinatorial conformal modules of the feature graphs with close viewpoints are close as well. We select four corners which are consistent among all the house images and separate the exterior face boundary into four segments, denoting as left, bottom, right, and top respectively. Applying our proposed method into these graphs, we can compute the conformal module of each image. Each image is associated with a conformal module, then the data is cast on the plane as shown in Fig. 4. It can be easily seen that there are three clusters of the conformal modules, which correspond to three major view positions. This means that the combinatorial conformal modules of the feature graphs with close viewpoints are close as well.

Fig. 3
figure3

Viewpoint detection experiment

Fig. 4
figure4

Conformal modules of house images

Posture detection

Another experiment is that we use CMU Graphics Lab Motion Capture Database for the application of the posture detection. The human actor wears 41 markers and makes a motion sequence. There are devices which capture the 3D position of the markers. In our experiment, we fix a virtual camera and project the markers onto the 2D image plane to analyze the result. Firstly, we use Delaunay triangulation to construct the graph. Then we compute the combinatorial conformal module of the graph. Four canonical postures are selected: standing, jumping, leaning back and walking as shown in Fig. 5. We project the posture onto the XOY,YOZ and ZOX plane respectively and calculate the corresponding Delaunay triangulation which is showed in the 3D posture scatter figures. We project the 3D posture onto the plane to find the most left vertex, the most bottom vertex, the most right vertex and the most top vertex as the corners. The proposed method is applied to compute the combinatorial conformal module of a posture graph. The combinatorial conformal module can be used to distinguish these postures. We use the proposed method to process the whole video sequence with 2751 frames and use K-nearest neighbor method to classify the postures. The correct rate of posture classification result is nearly 90%.

Fig. 5
figure5

Posture detection experiment

Conclusion

Graph matching has been intensively applied in computer vision for finding correspondences between two sets of features extracted from images. Most of the existing graph matching methods require high time complexity and are not usable to large graphs. Furthermore, these methods can hardly handle the circumstances where two graphs to be matched have large non-rigid transformations. In this work, we propose a novel graph matching and classification method by the combinatorial conformal module.

We generalize the classical Hodge theory to the combinatorial graph setting. The process is that we apply combinatorial Hodge theory to compute the combinatorial harmonic functions on graphs. By defining combinatorial Laplace matrices on graphs, the solution of harmonic functions is converted into solving linear Laplace matrices. The combinatorial harmonic 1-forms can give the combinatorial conformal invariant of a given graph, called as conformal module. According to the mathematical result, the conformal module can be used as the fingerprint of a given graph. Thus we can use the conformal module to do some matching work. Comparing to other graph matching methods, the proposed method can discover the combinatorial invariant of a given graph. Furthermore, the proposed method can tolerate the non-rigid transformation on a given graph. Due to considering the proposed method from the algebraic topology viewpoint, the computation of matching problem can be done in linear time. More importantly, our proposed method is adaptable to not only planar graphs but also graphs embedded in high genus surfaces.

In the future work, we purpose to explore the inexact graph matching method based on the conformal module and we hope the proposed method can be used to more applications in computer vision.

Abbreviations

GM:

Graph matching

References

  1. 1.

    F. Zhou, F. De la Torre, Factorized graph matching. IEEE Conference on Computer Vision and Pattern Recognition. 38, 127–134 (2012)

  2. 2.

    O. Duchenne, A. Joulin, J. Ponce, A graph-matching kernel for object categorization. International Conference on Computer Vision (2011), pp. 1792–1799

  3. 3.

    H. Jiang, S. X. Yu, D. R. Martin, Linear scale and rotation invariant matching. IEEE Trans. Pattern Anal. Mach. Intell. 33(7), 1339–1355 (2011)

  4. 4.

    J. Hays, M. Leordeanu, A. A. Efros, Y. Liu, Discovering texture regularity as a higher-order correspondence problem. European Conference on Computer Vision. (Springer, Berlin, Heidelberg, 2006), pp. 522–535.

  5. 5.

    U. Gaur, Y. Zhu, B. Song, A. Roychowdhury, A "string of feature graphs" model for recognition of complex activities in natural videos. IEEE International Conference on Computer Vision (ICCV), (IEEE, 2011), pp. 2595-2602

  6. 6.

    W. Brendel, S. Todorovic, Learning spatiotemporal graphs of human activities. IEEE International Conference on Computer Vision (2011), pp. 778–785

  7. 7.

    D. Conte, P. Foggia, C. Sansone, M. Vento, Thirty years of graph matching in pattern recognition. Int. J. Pattern Recognit. Artif. Intell. 18(03), 265–298 (2004)

  8. 8.

    F. Zhou, F. De la Torre, Deformable graph matching. IEEE Conference on Computer Vision and Pattern Recognition (2013), pp. 2922–2929

  9. 9.

    W. Zeng, J. Marino, K. Chaitanya Gurijala, X. Gu, A. Kaufman, Supine and prone colon registration using quasi-conformal mapping. IEEE Trans. Vis. Comput. Graph. 16(6), 1348–1357 (2010)

  10. 10.

    W. Zeng, Y. Zeng, Y. Wang, X. Yin, X. Gu, D. Samaras, 3d non-rigid surface matching and registration based on holomorphic differentials. European Conference on Computer Vision (Springer, Marseille, France, October 12–18, 2008), pp. 1–14

  11. 11.

    L. Lui, W. Zeng, S. Yau, X. Gu, Shape analysis of planar multiplyconnected objects using conformal welding. IEEE Trans. Pattern Anal. Mach. Intell. 36(7), 1384–1401 (2014)

  12. 12.

    X. D. Gu, S. T. Yau, Computational conformal geometry. (Somerville, Mass, USA: International Press, 2008)

  13. 13.

    J. R. Ullmann, An algorithm for subgraph isomorphism. J. ACM 23(1), 31–42 (1976)

  14. 14.

    L. P. Cordella, P. Foggia, C. Sansone, F. Tortorella, M. Vento, Graph matching: a fast algorithm and its evaluation. International Conference on Pattern Recognition. 2, 1582–1584 (1998)

  15. 15.

    L. P. Cordella, P. Foggia, C. Sansone, M. Vento, Fast graph matching for detecting cad image components. 15th International Conference on Pattern Recognition. 2, 1034–1037 (2000)

  16. 16.

    B. D. McKay, Practical graph isomorphism[J]. J. Symb. Comput. 60(1), 94–112 (2014)

  17. 17.

    C. Irniger, H. Bunke, Graph matching: filtering large databases of graphs using decision trees[C]. Graph-Based Representations in Pattern Recognition (2001), pp. 239-249

  18. 18.

    M. Lazarescu, H. Bunke, S. Venkatesh, Graph matching: fast candidate elimination using machine learning techniques. Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR). (Springer, Berlin, Heidelberg, 2000), pp. 236-245

  19. 19.

    E. M. Loiola, N. M. M. de Abreu, P.O. Boaventura-Netto, P. Hahn, T. Querido, A survey for the quadratic assignment problem. Eur. J. Oper. Res. 176(2), 657–690 (2007)

  20. 20.

    S. Umeyama, An eigendecomposition approach to weighted graph matching problems. IEEE Trans. Pattern Anal. Mach. Intell. 10(5), 695–703 (1988)

  21. 21.

    L. S. Shapiro, J. Michael Brady, Feature-based correspondence: an eigenvector approach. Image Vis. Comput. 10(5), 283–288 (1992)

  22. 22.

    T. Caelli, S. Kosinov, An eigenspace projection clustering method for inexact graph matching. IEEE Trans. Pattern Anal. Mach. Intell. 26(4), 515–519 (2004)

  23. 23.

    M. Leordeanu, M. Hebert, A spectral technique for correspondence problems using pairwise constraints. Tenth IEEE International Conference on Computer Vision. 2, 1482–1489 (2005)

  24. 24.

    H. Qiu, E. R. Hancock, Graph matching and clustering using spectral partitions. Pattern Recogn. 39(1), 22–34 (2006)

  25. 25.

    S. Gold, A. Rangarajan, A graduated assignment algorithm for graph matching[J]. IEEE Trans. Pattern Anal. Mach. Intell. 18(4), 377–388 (1996)

  26. 26.

    M. Zaslavskiy, F. Bach, J. P. Vert, A path following algorithm for the graph matching problem. IEEE Trans. Pattern Anal. Mach. Intell. 31(12), 2227–2242 (2009)

  27. 27.

    R. Zass, A. Shashua, Probabilistic graph and hypergraph matching. IEEE Conference on Computer Vision and Pattern Recognition(CVPR), (IEEE, 2008), pp. 1–8

  28. 28.

    M. Chertok, Y. Keller, Efficient high order matching. IEEE Transactions on Pattern Analysis and Machine Intelligence. 32(12), 2205-2215(2010)

  29. 29.

    O. Duchenne, F. Bach, I. S. Kweon, J. Ponse, A tensor-based algorithm for high-order graph matching. IEEE transactions on pattern analysis and machine intelligence. 33(12), 2383-2395(2011)

  30. 30.

    O. Schramm, Square tilings with prescribed combinatorics[J]. Israel Journal of Mathematics. 84(1–2), 97–118 (1993)

  31. 31.

    G. Guennebaud, B. Jacob, et al., Eigen v3. http://eigen.tuxfamily.org/index.php?title=Main_Page (2010)

Download references

Acknowledgements

The authors thank the editor and anonymous reviewers for their helpful comments and valuable suggestions.

Funding

Not applicable.

Availability of data and materials

Please contact author for data requests.

Author information

All authors take part in the discussion of the work described in this paper. These authors contributed equally to this work. Both authors read and approved the final manuscript.

Correspondence to Kun Qian.

Ethics declarations

Authors’ information

Jialing Zhang was born in Yunnan, P.R. China, in 1981. She received her Master of Science degree from Yunnan Normal University in 2006. She has been a faculty in Kunming University of Science and Technology since 2006. Her main research interests focus on Computational Geometry-Theory and Applications.

Kun Qian was born in Yunnan, P.R. China, in 1982. He received his M.S. degree in Mechanical and Electronic Engineering from Xidian University in China in 2009. He was a visiting scholar at the State University of New York at Stony Brook and University of Louisiana at Lafayette, US from January 2015 to January 2016. He is currently pursuing his Ph.D. degree in Kunming University of Science and Technology in China. His research interests are in the area of computational geometry, computer graphics and computer vision.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Keywords

  • Graph matching
  • Graph classification
  • Conformal module