Graph matching using conformal module

Abstract

Graph matching and classification play fundamental roles in computer vision. The computational complexity of the conventional method based on a spectrum method is high, which prevents it from handling large graphs in practice. This work proposes a novel framework for tackling the challenge by using conformal module. We apply the classical Hodge theory from differential manifold to the graph setting and compute the combinatorial conformal invariant of the graph, called as conformal module, which can be used as the fingerprint for the graph. The method is applicable for viewpoint classification and posture detection. The experimental results demonstrate the efficiency and efficacy of the proposed method.

1 Introduction

Graph matching has been widely used to solve a large variety of problems in computer vision. Fundamentally, graph matching aims at finding correspondences between two sets of features extracted from images. When matching or recognizing structural objects, graph matching does well on considering the pairwise node interactions [1]. Furthermore, when there are correspondences between two images, graph matching can finish many vision tasks such as object categorization [2], feature tracking [3], symmetry analysis [4], and posture recognition [5, 6].

In the past thirty years, extensive research has been carried out on graphing matching [7], however, there still exists several challenges in current methods. For example, almost all matching algorithms require high time complexity, so it is difficult to deal with large graphs. No methods have been provided to deal with graphs in 3D vision field, which means that graphs embedded in high genus surfaces cannot be matched using current approaches. Many techniques are proposed to solve graph matching in terms of rigid transformation, which can hardly handle the circumstance where two graphs to be matched have large non-rigid transformations. Although the third issue is considered in [8], the other two issues are still open. Our main contributions conclude as follows: (1) the proposed method can discover the combinatorial invariant of a given graph, (2) the proposed method can tolerate non-rigid transformation on a given graph, (3) the proposed method can solve the matching problem in linear time, (4) the proposed method can be applied on not only planar graphs, but also those embedded in high genus surfaces.

The main mathematical theory in our method is based on combinatorial Hodge theory. There is some work on surface registration [9, 10] and shape analysis [11] by the combinatorial Hodge theory. In the textbook [12], Gu et al. introduced the detailed theoretic treatment on combinatorial Hodge theory. These works focus on computing the conformal invariants and conformal mappings induced by the Riemannian structure of the input surfaces. By contrast, current work focuses on computing the conformal modules induced solely by the combinatorial structure of the graphs. The process is that we apply combinatorial Hodge theory to compute the combinatorial harmonic functions on graphs. The solution of harmonic functions is equivalent to solving combinatorial Laplace matrices. The combinatorial harmonic 1-forms can give the combinatorial conformal invariant of a given graph, called as conformal module, which can be used as the fingerprint of a given graph. So the conformal module can be used to deal with some matching problems. In this paper, we compute the conformal modules by the combinatorial structure of the given graphs for viewpoint classification and posture detection.

2 Previous works

Graph matching problem has been an active field in computer science and mathematics. There is a large literature for this topic in the past thirty years, so a thorough survey is beyond the scope of the current work. In the following, we only review the most related work. For more detailed treatment, we refer readers to [7]. There usually exists two categories of graph matching methods: exact matching and inexact matching.

Exact matching aims at graph isomorphism, where the interest is in an exact matching between nodes and edges. Most of approaches to find the graph isomorphism are based on Tree Search. Due to the brute Tree Search algorithm, some improvements are developed. Ullmann proposes one of the most important and popular tree search algorithms [13]. He prunes the unfruitful matches which are not consistent with current partial matching. Cordella suggests another interesting approach based on Tree Search [14, 15]. He provides a heuristic based on analysis of the sets of nodes adjacent to the ones already considered in current partial matching. Except for Tree Search, some other techniques are proposed. According to group theory, an approach was proposed by McKay [16]. He defines equivalence classes for graphs in terms of isomorphism and does the equality verification in O(n²). There are other methods based on machine learning techniques to speed up the matching against a large library of graphs [17, 18]. But they are used for filtering out the unmatched candidates not for matching.

Inexact matching focuses on mapping between graphs with weighted attributes on nodes and edges. The solution is a quadratic assignment problem and the optimization is NP-hard [19]. Looking for better optimization strategies is important for major research in graph matching. According to the relaxations of the constraints on a one-to-one mapping F between the nodes in both graphs [8], methods can be generally classified into several categories. The first is to approximate the constraints as an orthogonal one F^TF = I, which is called as spectral method. Under the orthogonal constraint, optimizing graph matching can be solved in closed form as an eigen value problem [20,21,22]. Furthermore, Leordeanu and Hebert relax F to be of unit length ${\left\Vert F\right\Vert}_2^2=1$ for handling more complex problems in computer vision [23]. The work in [24, 25] proposes a spectral method to decompose the graph into sub-graphs, where the process can be cast into a hierarchical framework and suitable for parallel computation. The second group of methods relaxes F to be a doubly stochastic matrix, the convex hull of F. Under this constraint, optimizing graph matching can be solved as a non-convex quadratic programming problem and a local optimization could be done within several proposed strategies. For instance, Gold and Rangarajan propose the graduated assignment algorithm for iteratively solving a series of linear approximations of the cost function in the form of Taylor expansion. The work of [26] proposes a path-following algorithm. Besides the optimization-based work, probabilistic frameworks are also useful for interpreting and solving graph matching [27]. Several work aims at higher-order tensor factorization [8, 28, 29], where higher order geometrical relations make graph matching invariant to rigid transformation. Due to a small increment in the order of relations induces a combination explosion of data, it is only suitable for very sparse graphs. Furthermore, this method cannot deal with non-rigid transformations.

3 Method

This section briefly introduces the theoretical background for combinatorial conformal module. We apply algebraic topology method to consider our matching problem. Algebraic topology focuses on studying topological problems by algebraic methods and uses simple linear algebraic methods to solve these problems. The detailed description of algebraic topology methodology can be seen in [12], where the authors consider these methods on surfaces with Riemann structure. In this paper, we apply algebraic topology method to the graph setting and consider combinatorial Hodge theory on graphs. Combinatorial Hodge theory can deduce a unique combinatorial conformal module for each graph, so it can be viewed as the fingerprint of a given graph.

We consider all objects on a three-connected planar graph G which is embedded on a topological surface S. A planar graph means that there are no edge crossings on G and different edges of the graph can only intersect at the endpoints. So each face of G can be viewed as a topological disk. We usually use a triple G = (V, E, F) to represent a graph, where V is the vertex set, E is the edge set and F is the face set. At the first step, we need to introduce some tools about computational topology. These similar concepts are defined on surfaces with Riemann structure in [12]. Here we use them on graphs.

The vertex set is denoted as

$$ V=\left\{{v}_1,{v}_2,\cdots, {v}_n\right\}. $$

(1)

Each oriented edge e is denoted as

$$ e=\left[{v}_i,{v}_j\right]\kern0.24em \mathrm{or}\kern0.24em e=-\left[{v}_j,{v}_i\right], $$

(2)

where v_i, v_j are its two endpoints.

A face f is represented as

$$ f=\left[{v}_{i_1},{v}_{i_2},\cdots, {v}_{i_k}\right], $$

(3)

where ${v}_{i_k}$ is its vertex and (i₁, i₂, ⋯, i_k) is any cyclic permutation of (1, 2, ⋯, k).

We call the vertices of G as 0-dimensional cells, the oriented edges of G as 1-dimensional cells and the faces of G as 2-dimensional cells. Their linear combinations can form a linear chain space. We will give some definitions on the chain space of a graph.

The k-dimensional chain group of a graph G = (V, E, F) is given by

$$ {C}_k\left(G,\mathrm{\mathbb{Z}}\right)=\left\{\sum \limits_{i=1}^{n_k}{\xi}_i{\varepsilon}_k^i,{\xi}_i\in \mathrm{\mathbb{Z}}\right\}, $$

(4)

where ${\varepsilon}_k^i$ denotes the ith k-dimensional cell of G_. We call each element of C_k(G, Z) as a k-chain of G.

Definition 1: The k-dimensional boundary operator ∂_k : C_k(G, ℤ) → C_k − 1(G, ℤ) on graph G is defined as

$$ {\partial}_k\left(\sum \limits_{i=1}^{n_k}{\xi}_i{\varepsilon}_k^i\right):= \sum \limits_{i=1}^{n_k}{\xi}_i{\partial}_k\left({\varepsilon}_k^i\right). $$

(5)

The boundary operator has the linear property. In this paper, we only consider the low dimensional boundary operator, for instance,

$$ {\partial}_0{v}_i=0,{\partial}_1\left[{v}_i,{v}_j\right]={v}_j-{v}_i,{\partial}_2\left[{v}_i,{v}_j,{v}_k\right]=\left[{v}_i,{v}_j\right]+\left[{v}_j,{v}_k\right]+\left[{v}_k,{v}_j\right]. $$

(6)

The k-chain in the kernel of ∂_k satisfies

$$ \mathit{\operatorname{Ker}}\ {\partial}_k=\left\{{\varepsilon}_k\in {C}_k\left(G,\mathrm{\mathbb{Z}}\right)|{\partial}_k{\varepsilon}_k=0\right\}. $$

(7)

We call each element in Ker ∂_k as a closed chain. The closed chains are closed loops and the boundaries of any 2-dimensional patches are called as boundary loops.

The k-chain in the image of ∂_k + 1 satisfies

$$ Img\ {\partial}_{k+1}=\left\{{\varepsilon}_k\in {C}_k\left(G,Z\right)|\exists {\varepsilon}_{k+1}\in {C}_{k+1}\left(G,Z\right),{\varepsilon}_k={\partial}_{k+1}{\varepsilon}_{k+1}\right\}. $$

(8)

Each element of Img ∂_k + 1 is called as an exact chain.

Property 1: The boundary of a boundary is empty:

$$ {\partial}_k\circ {\partial}_{k+1}\equiv 0. $$

(9)

The proof can be seen in [12].

From the above property, we can easily obtain that Img ∂_k + 1 is a sub-linear space of Ker ∂_k. The complementary space is called as the homology group of the graph.

Definition 2: Suppose ∂_k : C_k(G, ℤ) → C_k − 1(G, ℤ) is the k-dimensional boundary operator on a graph G. Ker ∂_k = {ε_k ∈ C_k(G, ℤ)| ∂_kε_k = 0} is the kernel of ∂_k. Img ∂_k + 1 = {ε_k ∈ C_k| ∃ε_k + 1 ∈ C_k + 1, ε_k = ∂_k + 1ε_k + 1} is the image of ∂_k + 1. The k-dimensional homology group of the graph G is defined as the quotient group, satisfying

$$ {H}_k\left(G,\mathrm{\mathbb{Z}}\right):= \frac{\mathit{\operatorname{Ker}}\ {\partial}_k}{Img\ {\partial}_{k+1}}. $$

(10)

Each element in H_k(G, Z) is called a k-dimensional homologous class. A closed k-chain ε_k ∈  ker ∂_k represents a homologous class, denoted as [ε_k] ∈ H_k(G, Z). Each element in the homologous class can form the basis of homology group. The similar algorithm in [12] is used to compute the homology basis of a discrete surface. Figures 1 and 2 show the homology basis of two graphs with different topology.

Fig. 1

Homology group basis for a graph embedded on a genus one surface

Fig. 2

Homology group basis for a graph embedded on a genus two surface

In order to do computation on graphs, we define the cohomology group which is the dual of the homology group. In fact, cohomology group is the space of linear functions defined on homology group. The following discussions are considered in parallel because all the objects are dual of the above counterparts such as co-chain and exterior differential operator.

The k-dimensional co-chain space of the graph G is a linear function defined on k-dimensional chain space and given by

$$ {C}^k\left(G,\mathrm{\mathbb{R}}\right)=\left\{{\omega}^k:{C}_k\left(G,Z\right)\to \mathrm{\mathbb{R}}\right\}. $$

(11)

We call each element in C^k(G, R) as a k-form.

The dual of the boundary operator ∂ is called as the exterior differential operator d.

Definition 3: The exterior differential operator d_k : C^k(G, ℝ) → C^k + 1(G, ℝ) is defined as follows:

$$ {d}_k{\omega}^k\left({\varepsilon}_{k+1}\right):= {\omega}^k\left({\partial}_{k+1}{\varepsilon}_{k+1}\right), $$

(12)

where ω^k ∈ C^k(G, R), ε_k + 1 ∈ C_k + 1(G, Z).

Corresponding to closed chain, we have that any k-form in the kernel of d_k satisfies

$$ \mathit{\operatorname{Ker}}\ {d}_k=\left\{{\omega}^k\in {C}^k\left(G,\mathrm{\mathbb{R}}\right)|{d}_k{\omega}^k=0\right\}. $$

(13)

We call each element in Ker d_k as a closed k-form.

Similarly, any k-form in the image of d_k satisfies

$$ Img\ {d}_{k-1}=\left\{{\omega}^k\in {C}^k\left(G,R\right)|\exists {\omega}^{k-1}\in {C}^{k-1}\left(G,R\right),{\omega}^k={d}_{k-1}{\omega}^{k-1}\right\}. $$

(14)

We call each element in Ker d_k as an exact k-form.

From Property 1, we obviously obtain the following property.

Property 2: The exterior differential operator d_k : C^k(G, ℝ) → C^k + 1(G, ℝ) is a linear operator with the property

$$ {d}_k\circ {d}_{k-1}\equiv 0. $$

(15)

According to property 2, we have that exact forms are closed. Therefore, Img d_k − 1 is a sub-linear space of Ker d_k. The complementary space is defined as the cohomology group of the graph.

Definition 4: Suppose the linear operator d_k : C^k(G, ℝ) → C^k + 1(G, ℝ) is the exterior differential operator on a graph G. Ker d_k = {ω^k ∈ C^k(G, ℝ)| d_kω^k = 0} is the kernel of d_k. Img d_k − 1 = {ω^k ∈ C^k(G, R)| ∃ω^k − 1 ∈ C^k − 1(G, R), ω^k = d_k − 1ω^k − 1} is the image of d_k − 1. The k-dimensional cohomology group of the graph G is defined as the quotient group, satisfying

$$ {H}^k\left(G,R\right):= \frac{\mathit{\operatorname{Ker}}\ {d}_k}{Img\ {d}_{k-1}}. $$

(16)

A graph G embedded on a topological surface S can induce a CW-cell decomposition. The Poincare dual of G is denoted as $\overline{G}$ which is also a graph embedded on S such that each vertex v_i of G corresponds to a face ${\overline{v}}_i$ of $\overline{G}$ and each face f_j of G corresponds to a vertex ${\overline{f}}_j$ of $\overline{G}$. Furthermore, each oriented edge e of G corresponds to an oriented edge $\overline{e}$ on $\overline{G}$ as follows: if e = [v_i, v_j], $\overline{e}={\overline{v}}_i\cap {\overline{v}}_j$. Furthermore, if the left face of e is f_l and the right face of e is f_r, $\overline{e}=\left[{\overline{f}}_r,{\overline{f}}_l\right]$.

Definition 5: Suppose a graph and its dual are denoted as G and $\overline{G}$ respectively. The Hodge star operator ${}^{\ast }:{C}^k\left(G,\mathrm{\mathbb{R}}\right)\to {C}^{2-k}\left(\overline{G},\mathrm{\mathbb{R}}\right)$ is a linear operator, satisfying

$$ {}^{\ast}\omega \left(\overline{\varepsilon}\right):= \omega \left(\varepsilon \right), $$

(17)

where ω ∈ C^k(G, R), ε ∈ C_k(G, Z), $\overline{\varepsilon}\in {C}_{2-k}\left(\overline{G},Z\right)$, and ${}^{\ast}\omega \in {C}^{2-k}\left(\overline{G},R\right)$.

Combining definition 3 and definition 5, we can obtain another linear operator σ_k : C^k(G, R) → C^k − 1(G, R), satisfying

$$ {\sigma}_k{=}^{\ast }{d_k}^{\ast }. $$

(18)

The operator σ_k can be viewed as the adjoint of the exterior differential operator, called as the co-differential operator. Combining operator d_k and σ_k, we can define the Laplace operator.

Definition 6: Suppose d_k : C^k(G, ℝ) → C^k + 1(G, ℝ) is the exterior differential operator on a graph $G.{}^{\ast }:{C}^k\left(G,\mathrm{\mathbb{R}}\right)\to {C}^{2-k}\left(\overline{G},\mathrm{\mathbb{R}}\right)$ is the Hodge star operator on G. σ_k = ^∗d_x^∗ : C^k(G, R) → C^k − 1(G, R) is the co-differential operator on G. The Laplace operator Δ : C^k(G, R) → C^k(G, R) is defined as

$$ \varDelta := {\sigma}_{k+1}{d}_k+{d}_{k-1}{\sigma}_k. $$

(19)

Obviously, the Laplace operator is symmetric, that is, for any ξ, η ∈ C^k(G, R),

$$ \left(\varDelta \zeta, \eta \right)=\left(\zeta, \varDelta \eta \right). $$

(20)

Furthermore,

$$ \left(\varDelta \zeta, \zeta \right)\ge 0. $$

(21)

We mainly consider functions whose values are equal to zero with the action of the Laplace operator. These functions have inspiring mathematical and physical properties. In this paper, we will compute these functions on the graph setting. Based on the combinatorial structure of a graph, the solution of harmonic functions is equivalent to solving combinatorial Laplace matrices on the graph. Finally, these combinatorial harmonic functions can give out the combinatorial conformal invariant of a given graph, called as conformal module, which can be used as the fingerprint of a given graph. So, the conformal module can be applied into matching applications on graphs.

Definition 7: Suppose d_k : C^k(G, ℝ) → C^k + 1(G, ℝ) is the exterior differential operator on a graph G. σ_k = ^∗d_x^∗ : C^k(G, R) → C^k − 1(G, R) is the co-differential operator on G. Δ = σ_k + 1d_k + d_k − 1σ_k : C^k(G, R) → C^k(G, R) is the Laplace operator on G. A k-form ω ∈ C^k(G, ℝ) is called to be harmonic, if

$$ \varDelta \omega =0. $$

(22)

Since (Δω, ω) = |d_kω|² + |σ_kω|² = 0, we can easily have the following property.

Property 3: A harmonic k-form ω ∈ C^k(G, ℝ) satisfies

$$ \left\{\begin{array}{ccc}{d}_k\omega & =& 0\\ {}{\sigma}_k\omega & =& 0.\end{array}\right. $$

(23)

From the physical viewpoint, the exterior differential operator d_k is the curl operator and the co-differential operator σ_k is the divergence operator. So a harmonic k-form means that it is both curl-free and divergence-free.

The group of all the harmonic k-forms is denoted as ${H}_{\Delta}^k\left(G,\mathrm{\mathbb{R}}\right)$, namely

$$ {H}_{\Delta}^k\left(G,\mathrm{\mathbb{R}}\right)=\mathit{\operatorname{Ker}}\ {d}_k\cap \mathit{\operatorname{Ker}}\ {\sigma}_k. $$

(24)

According to Hodge theory, any a k-form can be decomposed into an exact form, a co-exact form and a harmonic form.

Theorem 1: Suppose ω is a k-form. d_k − 1 : C^k − 1(G, ℝ) → C^k(G, ℝ) is the exterior differential operator on a graph G. σ_k = ^∗d_x^∗ : C^k(G, R) → C^k − 1(G, R) is the co-differential operator on G. Then there is a unique (k − 1)-form α, (k + 1)-form β and a harmonic k-form γ, such that

$$ \omega ={d}_{k-1}\alpha +{\sigma}_{k+1}\beta +\gamma; $$

(25)

moreover, such kind of decomposition is unique.

The detailed proof can be checked in [12].

For example k = 2, suppose G is embedded on a closed surface with genus g.{γ₁, γ₂, ⋯, γ_2g} is the generator of 1-dimensional homology group H₁(G, Z). Let ω₁, ω₂ be two harmonic 1-forms with respect to the homology equivalence class, namely

$$ {\int}_{\gamma_k}{\omega}_1={\int}_{\gamma_k}{\omega}_2,k=1,2,\cdots, 2g. $$

(26)

Due to the linear property, we have

$$ {\displaystyle \begin{array}{ccccc}d\left({\omega}_1-{\omega}_2\right)& =& d{\omega}_1-d{\omega}_2& =& 0,\\ {}\sigma \left({\omega}_1-{\omega}_2\right)& =& {\sigma \omega}_1-{\sigma \omega}_2& =& 0.\end{array}} $$

(27)

So ω₁ − ω₂ is harmonic and ${\int}_{\gamma_k}\left({\omega}_1-{\omega}_2\right)={\int}_{\gamma_k}{\omega}_1-{\int}_{\gamma_k}{\omega}_2=0.\left({\omega}_1-{\omega}_2\right)$ is exact. There is a function f : G → R such that ω₁ − ω₂ = df. According to the maximum principle and the empty boundary, f has no extremal value and is constant. That means the uniqueness holds: ω₁ − ω₂ = df = 0.

Suppose ω ∈ H¹(G, R) is closed and f : G → R is a smooth function. So ω + df and ω lie in the same cohomology class. The solution of σ(ω + df) = 0 is equivalent to the solution of Poisson equation:

$$ \varDelta f=-\sigma \omega . $$

(28)

In order to compute harmonic functions on a graph, we need define the combinatorial Laplace matrix.

Definition 8: Suppose G = {v₁, v₂, ⋯, v_n} is a planar graph. The combinatorial Laplace matrix Δ = (δ_ij) on G is defined as

(29)

where v_i~v_j means two vertices are adjacent and d(v_i) is the degree of v_i in the graph. According to theorem 1, the following Corollary is straightforward.

Corollary: The group of all harmonic k-forms on G is isomorphic to the k-dimensional cohomology group, that is

$$ {H}_{\Delta}^k\left(G,\mathrm{\mathbb{R}}\right)\cong {H}^k\left(G,\mathrm{\mathbb{R}}\right). $$

(30)

Suppose ω₁ and ω₂ are cohomological k-forms, then there is a (k − 1)-form η, such that ω₁ = ω₂ + dη. Assume the Hodge decomposition of ω₁ is

$$ {\omega}_1=d{\alpha}_1+{\delta \beta}_1+{\gamma}_1=d{\alpha}_1+{\gamma}_1. $$

(31)

dω₁ = 0, dδβ₁ = 0, so δβ₁ = 0. Directly, we get

$$ {\omega}_2=d\left({\alpha}_1-\eta \right)+{\gamma}_1. $$

(32)

That means two cohomological forms share the same harmonic form. Therefore, each cohomological class in H^k(G, R) corresponds to a unique harmonic form in ${H}_{\Delta}^k\left(G,R\right)$, which establishes the isomorphism.

Combinatorial Conformal Invariants: Suppose a graph G is embedded on a genus g surface S. ω is a combinatorial harmonic 1-form on G and γ is a closed loop. We can choose a tubular neighborhood of γ,

$$ N\left(\gamma \right):= \left\{\alpha \in G|\alpha \cap \gamma \ne \varnothing \right\}. $$

(33)

We can periodically embed N(γ) to the plane. Each period is represented as a complex number, denoted as ∫_γω.

We can choose a canonical homology group basis in H₁(G, Z) as {a₁, b₁, a₂, b₂, ⋯, a_g, b_g}. Namely, they satisfy the following conditions

$$ {a}_i\cdot {b}_i=1,{a}_i\cdot {a}_j=0,{b}_i\cdot {b}_j=0, $$

(34)

where a ⋅ b represents the algebraic intersection number. Suppose a closed differential 1-form is ω. The integral of ω on a_i

$$ {\int}_{a_1}\omega, {\int}_{a_2}\omega, \cdots, {\int}_{a_g}\omega $$

(35)

is called as the a-period of ω. Similarly,

$$ {\int}_{b_1}\omega, {\int}_{b_2}\omega, \cdots, {\int}_{b_g}\omega $$

(36)

is called the b-period of ω.

For all a_i, slicing the surface S to get S_i, the boundary of S_i is $\partial {S}_i={a}_i^{+}-{a}_i^{-}$. We construct a function f_i such that

$$ {f}_i\left({v}_k\right)=\left\{\begin{array}{c}1,{v}_k\in {a}_i^{+}\\ {}0,{v}_k\in {a}_i^{-}\\ {}\mathrm{random},\mathrm{otherwise}\end{array}\right., $$

(37)

and denote α_i = df_i.

For all b_j, slicing the surface S to get S_j, the boundary of S_j is $\partial {S}_j={b}_j^{+}-{b}_j^{-}$. We construct a function g_j such that

$$ {g}_j\left({v}_k\right)=\left\{\begin{array}{c}1,{v}_k\in {b}_j^{+}\\ {}0,{v}_k\in {b}_j^{-}\\ {}\mathrm{random},\mathrm{otherwise}\end{array}\right., $$

(38)

and denote β_j = dg_j.

The combinatorial period matrix of G is as following:

$$ Mod(G)={\left(\underset{a_i}{\int }{\alpha}_j\right)}^{-1}\left(\underset{b_k}{\int }{\beta}_l\right). $$

(39)

For genus one case, the graph G can be embedded into the plane as a parallelogram. The combinatorial period matrix Mod(G) represents the shape of the parallelogram. For high genus graph G, the combinatorial period matrix is the generalization. The combinatorial period matrix can be used as the fingerprint of the embedded graph G.

One face f₀ on the graph G is selected as the unique exterior face. Four vertices on the boundary of f₀ are chosen, denoted as {v₀, v₁, v₂, v₃}, sorted counter clockwisely. The graph G with four corner vertices is called as a topological quadrilateral. The corner vertices divide the boundary of the exterior face into four segments, denoted as left, top, right, and bottom segments, such that v₀ is the left lower corner. Then we can compute a harmonic function u : G → ℝ with Dirichlet boundary condition, namely

$$ \left\{\begin{array}{ccc}u\left({v}_i\right)& =& 1,{v}_i\in \mathrm{top},\\ {}u\left({v}_i\right)& =& 0,{v}_i\in \mathrm{bottom},\\ {}\Delta u\left({v}_i\right)& =& 0,\mathrm{otherwise}.\end{array}\right. $$

(40)

In fact, the harmonic 1-form ω is the exterior differential of u, ω = du.

Theorem 2: Suppose G is a topological quadrilateral which is a planar graph with an exterior face f₀ and four corner vertices {v₀, v₁, v₂, v₃}. The harmonic 1-form maps the topological quadrilateral to a rectangle with width w and height h, then the conformal module of the topological quadrilateral is

$$ Mod\left(G,\left\{{v}_0,{v}_1,{v}_2,{v}_3\right\}\right):= \frac{h}{w}. $$

(41)

This theorem can be proven by an extremal length theory. More detailed proof is shown in [30].

So, there is a conformal mapping ϕ which maps a topological quadrilateral into a canonical rectangle and maps four vertices {v₀, v₁, v₂, v₃} into four corner vertices of the rectangle. The ratio of width and height is the conformal module. Four vertices {v₀, v₁, v₂, v₃} divide the boundary of the topological quadrilateral into four segments, denoted as Γ₀, Γ₁, Γ₂, Γ₃ and ∂Γ_k = v_k + 1 − v_k. Solving two Dirichlet equations, there are two harmonic functions f₁, f₂ : G → R which satisfy the Dirichlet boundary and the Neumann boundary:

$$ \left\{\begin{array}{ccc}\varDelta {f}_1& =& 0,\\ {}{\left.{f}_1\right|}_{\Gamma_0}& =& 0,\\ {}{\left.{f}_1\right|}_{\Gamma_2}& =& 1,\\ {}{\left.\frac{\partial {f}_1}{\partial n}\right|}_{\Gamma_1\cup {\Gamma}_3}& =& 0;\end{array}\right. $$

(42)

$$ \left\{\begin{array}{ccc}\varDelta {f}_2& =& 0,\\ {}{\left.{f}_2\right|}_{\Gamma_3}& =& 0,\\ {}{\left.{f}_2\right|}_{\Gamma_1}& =& 1,\\ {}{\left.\frac{\partial {f}_2}{\partial n}\right|}_{\Gamma_0\cup {\Gamma}_2}& =& 0.\end{array}\right. $$

(43)

where n is the normal vector. The gradient fields ∇f₁ and ∇f₂ are mutually vertical lines but are not conjugate to each other. A constant λ is necessary for the conformal mapping ϕ with $\phi =\lambda {f}_2+\sqrt{-1}{f}_1$. So λ is the width of the canonical rectangle. Due to the harmonic function with conformal invariant property, there is a mapping energy

$$ E\left({f}_1\right)={\int}_G{\left|\nabla {f}_1\right|}^2 dA=\lambda, $$

(44)

and

$$ \phi =E\left({f}_1\right){f}_2+\sqrt{-1}{f}_1. $$

(45)

4 Experimental results and discussions

All experiments are conducted on a Windows 7 platform, with a single 2.3-GHz Intel CPU, 16-GB RAM memory. All algorithms have been developed in generic C++, compiled using Visual Studio 2010. The sparse linear systems are solved using the numerical library Eigen [31].

The experimental work is to apply the combinatorial Hodge theory to compute the harmonic functions on the graph setting and use these harmonic functions to get the combinatorial conformal module of a given graph. The combinatorial conformal module can be viewed as the fingerprint of a graph. For each image, we make a feature graph for it. Based on the combinatorial conformal modules of graphs, matching work can be done by finding correspondences between two sets of features extracted from images. We test the proposed method in the following applications.

4.1 Viewpoint detection

Figure 3 illustrates the experiment for viewpoint detection by the combinatorial conformal modules of graphs. We use the house images with features from CMU house datasets. Although all the images are the same building with the same feature, the combinatorial structure of the Delaunay triangulation varies according to the viewpoint change. If the viewpoints are close to each other, the corresponding combinatorial structure of the feature graph is similar. We expect that the combinatorial conformal modules of the feature graphs with close viewpoints are close as well. We select four corners which are consistent among all the house images and separate the exterior face boundary into four segments, denoting as left, bottom, right, and top respectively. Applying our proposed method into these graphs, we can compute the conformal module of each image. Each image is associated with a conformal module, then the data is cast on the plane as shown in Fig. 4. It can be easily seen that there are three clusters of the conformal modules, which correspond to three major view positions. This means that the combinatorial conformal modules of the feature graphs with close viewpoints are close as well.

Fig. 3

Viewpoint detection experiment

Fig. 4

Conformal modules of house images

4.2 Posture detection

Another experiment is that we use CMU Graphics Lab Motion Capture Database for the application of the posture detection. The human actor wears 41 markers and makes a motion sequence. There are devices which capture the 3D position of the markers. In our experiment, we fix a virtual camera and project the markers onto the 2D image plane to analyze the result. Firstly, we use Delaunay triangulation to construct the graph. Then we compute the combinatorial conformal module of the graph. Four canonical postures are selected: standing, jumping, leaning back and walking as shown in Fig. 5. We project the posture onto the XOY,YOZ and ZOX plane respectively and calculate the corresponding Delaunay triangulation which is showed in the 3D posture scatter figures. We project the 3D posture onto the plane to find the most left vertex, the most bottom vertex, the most right vertex and the most top vertex as the corners. The proposed method is applied to compute the combinatorial conformal module of a posture graph. The combinatorial conformal module can be used to distinguish these postures. We use the proposed method to process the whole video sequence with 2751 frames and use K-nearest neighbor method to classify the postures. The correct rate of posture classification result is nearly 90%.

Fig. 5

Posture detection experiment

5 Conclusion

Graph matching has been intensively applied in computer vision for finding correspondences between two sets of features extracted from images. Most of the existing graph matching methods require high time complexity and are not usable to large graphs. Furthermore, these methods can hardly handle the circumstances where two graphs to be matched have large non-rigid transformations. In this work, we propose a novel graph matching and classification method by the combinatorial conformal module.

We generalize the classical Hodge theory to the combinatorial graph setting. The process is that we apply combinatorial Hodge theory to compute the combinatorial harmonic functions on graphs. By defining combinatorial Laplace matrices on graphs, the solution of harmonic functions is converted into solving linear Laplace matrices. The combinatorial harmonic 1-forms can give the combinatorial conformal invariant of a given graph, called as conformal module. According to the mathematical result, the conformal module can be used as the fingerprint of a given graph. Thus we can use the conformal module to do some matching work. Comparing to other graph matching methods, the proposed method can discover the combinatorial invariant of a given graph. Furthermore, the proposed method can tolerate the non-rigid transformation on a given graph. Due to considering the proposed method from the algebraic topology viewpoint, the computation of matching problem can be done in linear time. More importantly, our proposed method is adaptable to not only planar graphs but also graphs embedded in high genus surfaces.

In the future work, we purpose to explore the inexact graph matching method based on the conformal module and we hope the proposed method can be used to more applications in computer vision.