Research Interests & Experience
· Computer Graphics
Ø Digital Geometry Processing (07/2005-present)
§ Discrete differential geometry (Laplacian differential framework, PDE geometric modeling, Gradient field manipulation, Mean-value manifold operator, Harmonic field, Discrete Exterior Calculus)
§ Modeling, Mesh deformation, Surface parameterization, Texture mapping, Mesh segmentation, Surface fairing / smoothing
§ My Contributions: Model Transduction, Mean-Value Manifold Operator, Convex-Hull Cross-Parameterization, Angle-Based Feature Sensitive Metric for Sketch-Based Mesh Segmentation, Consistent Correspondence between Arbitrary Manifold Surfaces, etc.
§ Supported by National High Technology Program (China's 863 program, Project No. 2005AA114130) and National Natural Science Foundation of China (NSFC No. 60675012)
Ø Interactive Techniques. (10/2005-present)
§ MeshLive: an extensible graphics toolkit for digital geometry processing (based on C++, QT, CGAL, OpenGL, STL, etc.)
§ SketchLive: a sketch-based interactive framework for mesh segmentation and editing.
§ CorrespondenceLive: consistent correspondence between arbitrary manifold surfaces
Ø (Crowd) Character Animation, Physically Based Simulation, Visualization
· Computer Vision
Ø Theory and Application of 3D Motion Capture System Based on Stereo Vision Methods. (11/2004-12/2005)
§ A key project of National High Technology Program (China's 863 program) (Project No. 2004AA119050, ￥5,000,000)
§ We developed a robust 3D motion capture system, including multi-camera synchronization and calibration, video collection, markers detecting and tracking, 3D reconstruction, motion editing, character animation, user interface, etc.
§ Motion data processing, such as model editing, motion tracking, character animation, and user interface.
Ø Facial Image Composition Based on Active Appearance Model (AAM). (12/2006-09/2007)
§ An easy-to-use framework for facial image composition, which can automatically exchange the source image's face or facial features onto the target image.
§ Poisson image editing and Matting, Statistical analysis of shape (principal component analysis, SVM), Training and searching of Active Appearance Model, Image warp based on thin-plate splines.
1、Global and Local Isometry-Invariant Descriptor for 3D Shape Comparison and Partial Matching based on Manifold Harmonics (“Shape-DNA”).
In this paper, based on manifold harmonics (“Shape-DNA”), we propose a novel framework for 3D shape similarity comparison and partial matching. First, we propose a novel symmetric mean-value representation to robustly construct high-quality manifold harmonic bases on nonuniform-sampling meshes. Then, based on the manifold harmonic bases constructed, a novel shape descriptor is presented to capture both of global and local features of 3D shape. This feature descriptor is isometry-invariant, i.e., invariant to rigid-body transformations and non-rigid bending. After characterizing 3D models with the shape features, we perform 3D retrieval with a up-to-date discriminative kernel. This kernel is a dimension-free approach to quantifying the similarity between two unordered feature-sets, thus especially suitable for our high-dimensional feature data. Experimental results show that our framework can be effectively used for both comprehensive comparison and partial matching among non-rigid 3D shapes.
Figure Characterize the shape features both globally and locally.
2、3D Face Tracking and Editing in Real-Time (Depth) Video
This project presents a novel approach to tracking 3D face in real time. Large-angle motions are carefully considered. Then, we propose an interesting application: exchanging/editing the face in real-time video. The high-quality results are achieved and can be used to web-based entertainment scenes.
Figure 3D Face Tracking and Editing in Real-Time (Depth) Video.
3、3D Human-Body Pose Reconstruction and Action Analysis in Real-Time Video
The action analysis is an important topic in video-based applications. In this project, inspired by Hornung's method, we work on 3D human-body pose reconstruction and action analysis in real-time video. First, a 3D human-body template is used to reconstruct the 3D human pose in real time. Then, we track the human motion and analyze his/her actions. Some interesting applications are developed.
Figure Inspired by Hornung's method, we work on 3D human-body pose reconstruction and action analysis in real-time video.
This paper proposes a novel method, called model transduction, to directly transfer pose between different meshes, without the need of building the skeleton configurations for meshes. Given a source mesh and a target mesh, a correspondence between two meshes is established from only a few pairs of markers. Then the pose of the source mesh is directly transferred to the target mesh while preserving the surface details of the target mesh. Different from previous retargetting methods, model transduction does not require an extra reference source mesh to obtain source deformation, and provides more flexibility to handle deformation operation with better control. Our approach is numerically efficient, as the solution to the optimization problem can be obtained by fast solving a sparse linear system. Experimental results show that model transduction can successfully transfer both complex skeletal structures and subtle skin deformations.
Figure (a) Deformation transfer [Sumner:2004:SIGGRAPH] cannot produce satisfying results when the source and target have different reference poses. (b) With model transduction, a lion successfully imitates a cat's pose even if the reference source mesh is absent.
Figure With the model transduction method, (a) an old man directly imitates a young man's expression; (b) the muscular man model directly imitates the fetus model's pose.
5、Mean-Value Manifold Operator
The mean-value manifold operator is a shape-preserving operator defined on manifold meshes (including detailed and highly irregular meshes), and furnishes a variety of processing applications. By encoding and decoding both parameterized topology information and detailed geometry information, this operator performs detail (or parameterization) preserving operations for manifold meshes. More importantly, the mean-value manifold operator proposed in this paper aims at building a weak vector field (quasi-scalar field) while holding the linear property. Together with the nice properties (proper encoding weights and small ratios), our editing system is effective and stable when the handles are under large angle transformations or moved rapidly by the user.
Figure Mean-value manifold operator and its application for mesh editing.
6、Angle-Based Feature Sensitive Metric and the sketch-based mesh segmentation framework
Meaningful mesh segmentation plays a more and more important role in various graphics applications, such as texture mapping, shape retrieval, and high-quality metamorphosis. This paper proposes a sketch-based interactive framework for real-time mesh segmentation. With an easy-to-use tool, the user can freely segment a 3D mesh while needing little effort or skill. In order to meaningfully segment the mesh, two dimensionless feature sensitive metrics are proposed, which are independent of the model and part size. We show that these metrics give the clear physical meaning to illustrate discrete differential geometric features, such as the curvature tensor and the curve length of Gaussian image. Finally, we discuss three kinds of boundary smoothing methods, and present two fast topology-invariant, geometry-invariant adjustment algorithms based on convex hull features and angle features. Compared with previous methods, our method can easily achieve multi-level segmentation in just one session.
Figure Angle-Based Feature Sensitive Metric and the sketch-based mesh segmentation framework
7、Convex Hull Cross-Parameterization
Cross-parameterization (or consistent parameterization, inter-surface mapping), is a useful tool to transform models into compatible meshes for the convenience of many geometry processing applications, such as metamorphosis, texture transferring, etc. Existing cross-parameterization methods suffer from several problems: slow speed, tedious user interaction, low robustness, hardness to obtain meaningful inter-surface mapping. These drawbacks reduce this technique's usefulness. In this paper, we introduce a novel scheme to solve these problems. First, based on our scale-independent feature sensitive metrics, we present an easy-to-use tool to produce meaningful mesh segmentation as cross-parameterization's prerequisite. Then we construct convex hull for each part of segmented meshes, and adopt our convex hull cross-parameterization algorithm to generate compatible meshes. Our scheme exploits the excellent properties of convex hull, e.g. good approximating ability and linear convex representation for interior vertices. Compared with previous approaches, our scheme is robust, fast, and convenient, thus suitable for common applications.
Figure Convex Hull Cross-Parameterization
8、Consistent Correspondence between Arbitrary Manifold Surfaces
We propose a novel framework for consistent correspondence between arbitrary manifold meshes. Different from most existing methods, our approach directly maps the connectivity of the source mesh onto the target mesh without needing to build intermediate domains and partitions for input meshes, thus effectively avoids dealing with unstable extreme conditions (e.g. complex boundaries or high genus). In this paper, firstly, a novel mean-value Laplacian fitting scheme is proposed, which aims at computing a shape-preserving (conformal) correspondence directly in 3D-to-3D space, efficiently avoiding local optimum caused by the nearest-point search, and achieving good results even with only a few marker points. Secondly, we introduce a vertex relocation and projection approach, which refines the initial fitting result in the way of local conformity. Furthermore, we provide a fast and effective approach to automatically detect critic points in the context of consistent correspondence.
(a) (b) (c) (d) (e)
Figure Consistent Correspondence between Arbitrary Manifold Surfaces
9、Theory and Application of 3D Motion Capture System Based on Stereo Vision Methods
A key project of National High Technology Program (China's 863 program) (Project No. 2004AA119050, ￥5,000,000)
Develop a robust 3D motion capture system, including multi-camera synchronization and calibration, video collection, markers detecting and tracking, 3D reconstruction, motion editing, character animation, user interface, etc.
Motion capture data processing, such as model editing, motion tracking, character animation, and user interface.
Figure Theory and Application of 3D Motion Capture System Based on Stereo Vision Methods
Meanwhile, an efficient skeleton-free pose retargetting method is presented. Firstly, by using our novel skeleton-free skinning technique, the overall pose of marker points from motion capture (Mocap) is retargetted to a mesh without needing to build skeleton structures for these points. The computation complexity of our technique is irrelative to the number of retro-reflective markers, as the markers just serve as the constraints of the solution system in a uniform least squares sense. Then, we show that our method is very useful for pose correction after various mesh editing operations. Our approach is numerically efficient, as the solution to the optimization problem can be obtained by fast solving a sparse linear system.
Figure An Efficient Skeleton-Free Mesh Deformation Method With Motion Capture Data
10、Facial Image Composition Based on Active Appearance Model
In this paper, based on Active Appearance Model (AAM), we present an easy-to-use framework for facial image composition, which can automatically exchange the source image's face or facial features onto the target image. The manual interaction is simple and the user only needs to input semantic information of ROI (region of interest) to be exchanged, such as 'face' or 'eyes'. Our framework mainly consists of two steps: model fitting and component compositing. Model fitting is designed to interpret each input image and obtain a synthesized model face of the image. Then by using component compositing, visual pleasing result is generated by solving Poisson equation with the boundary condition, produced automatically from model fitting. Furthermore, we propose a solution for eliminating the artifacts when part of the target face is occluded by hair, glasses, etc. The visually satisfactory results demonstrate the effectiveness of our facial image composition system.
Figure Facial Image Composition (Automatically Exchange Faces) Based on Active Appearance Model
11、Discrete Exterior Calculus for Mesh Deformation (under working)
Geometry plays an important role in science, ranging from computer graphics, computer vision to various physical theories (such as general relativity, electromagnetism, and solid & fluid mechanics). Exterior Calculus of Differential Forms gives the most elegant description of the geometric nature of these rules. For instance, the formulas of gradient, curl, divergence as well as the theorems of Green, Ostrovski-Gauss and Stokes can be uniformly described with the Exterior Calculus Form on manifold. However, Discrete Exterior Calculus (DEC) so far has only a few of initial applications in computer science and physics. That is, it is still a very new direction and has a lot of potentials for the deep discovery in these fields. Currently, my work focuses on mesh deformation with DEC, which aims at holding many nice deformation properties such as volume-preserving, detail-preserving and avoiding self-intersections.
Figure Discrete Exterior Calculus for Mesh Deformation