1. Objectifs principaux du travail (env. 15 lignes)

Motion capture is the process that consists of measuring and recording human motion, in view of applying it to a 3D model, and this technique has become increasingly popular in the animation and video game area over the past years. If the process leading from motion capture to model animation is efficient, the output animation will be fluid, human-like and realistic, thus presenting an enormous gain on physically-based animation. Because magnetic motion capture allows too little freedom of movement for the performer, optical motion capture is now the standard (with or without markers). Many of all-in-one motion capture systems are available on the market. They integrate the hardware and the software for processing the captured data, thus providing an animated model that will subsequently be applied to the end-user’s model.

The main problem is to infer the state of the model that would best fit the data. The quality of this fit will determine the quality of the resulting animation.

Determining the posture of the model in each frame of the sequence is what we will refer to as fitting the model to the data. Models can range from a skeleton to a complete humanoid, with muscles, fat and skin. The data can be acquired using an optical motion capture system or derived from video-sequences using computer vision techniques. In most cases, as the models have numerous DOFs in their parameters, there are many almost equivalent solutions to the fitting problem. In order to reduce the number of possible solutions, we constrain the model to reduce the search space. We will take advantage of bio-mechanical constraints such as the joint angle limitations, the inter-dependence of the joint movements, and contact constraints, just to mention a few.

In the case of motion capture using reflective markers, we must also deal with the relative unreliability of the surface position of the 3D data. The surface of the actor (skin, clothes) is not tightly bound to the underlying muscles, fat, bones, and therefore has a relative movement of its own with respect to the joints. Therefore, all data observed on the surface (markers or contours) are necessarily subject to such variations, which we need to taking into account when performing the model to data fit. 
We intend to focus on the case of the shoulder, as it is generally agreed to be the most complex area of the human body when it comes to 3D modelling. At a later stage, we plan to extend our approach to the entire body.

Another problem to be addressed is fluidity of motion. If each frame posture is obtained through a fitting process, there is a chance that the overall animation will sometimes be jerky, especially if there are sudden accelerations in the gestures. It is therefore necessary to ensure continuity and smoothing from one frame to another, once more in the form of constraints applied to the model parameters.

2. Résultats obtenus durant l'année 2000 (juin à décembre, env. 40 lignes)

Before working on adding biomechanical constraints to our present humanoid model, we need to create the infrastructure that will allow us to fit a model to 3D dynamic data. In this view, we have created two libraries, as well as an application providing interfaces to these two libraries and visualisation of the results.

In order to make our work as re-usable as possible, we developed a generic fitting library that can be applied to numerous types of data, models, and rotation representations. More precisely, the library LibFittingGraph stores the model as a hierarchy of nodes, providing at all times the position in 3D space of these nodes, i.e. the current state of a hierarchy. Into this library, we can plug-in various node types. For our case, we have defined nodes of the ‘joint’ type and of the ‘transform’ type. Our hierarchy thus consists alternatively of transform nodes and joint nodes, each transform node applying transforms onto its child joint node (classical principle of a scene graph). A node of the 'transform' type carries all the transformations (presently, translations and rotations) to be applied to its direct children nodes. The nodes of the 'joint' type store their own co-ordinates in global 3D co-ordinates, thus allowing us to access the position of any node in space, at each step of the fitting process.

We can plug-in various types of transforms into the transform nodes. Currently implemented are Euler angles, axis-angles, exponential maps, quaternions, 3x3 rotation matrices and 4x4 rotation matrices. For each transformation type, we provide the calculation of Jacobian matrices and objective functions, in view of performing least-squares fitting of the hierarchical model to 3D data. Presently, all transformations on the lab's humanoids were performed using Euler angles. However this rotation representation yields numerous problems, and we will therefore switch to other representations, such as the exponential map [1].

Once we had a functioning library providing us with a generic representation of an articulated figure, we developed a library providing a least-squares fitting framework and interface, LibFitting, this library being itself based on a lower-level fitting library already available in the lab.

The LSQ fitting library uses the principles of inverse kinematics for fitting a model to an observation. Indeed, the problem is analogical to the IK problem, where we would like an end-effector to be positioned at certain co-ordinates in 3D space, and the output are the translational and rotational values of all the joints in the hierarchy. The solving of this problem is generally subject to constraints that need to be satisfied in order for a solution to be valid (see further down in this paragraph, as well as the 'Future Work' section). 

The distance from end-effector to the observation is the objective function, and for each parameter of the model (translation, rotation), we calculate its derivative with respect to the parameter. For this, we use an accelerated gradient calculation, as outlined in other lab publications [4]. For rotational joints, if we have an articulated structure as per the Figure below, with O_obs being the observation, O_E the end effector whose distance to the observation we are attempting to minimise, and O_j the 2 or 3-rotational joint around which we are performing a rotation, then we can express the derivative of the distance function d, with respect to each degree of freedom  as the product of two terms. The first is the derivative of the distance function with respect to the observation co-ordinates. The second term expresses the change induced by the rotation at joint j. Decomposing a rotational joint into a directional axis a and an angle value , i.e. the amount of rotation around this axis, we can rewrite the second term as the cross-product of the directional axis and the vector from joint j to the observation under the current rotation . 

In the case of translational joints, the translation actually represents the length of the bone segment. The initial bone lengths being T, the optimisation will modify this length by a coefficient t. Therefore, the derivative of a translational joint is the derivative of the distance function with respect to the observation co-ordinates multiplied by T.

Using these gradient calculations, neither pre-calculated derivatives nor prior knowledge of the hierarchy are necessary.

The lower-level least-squares fitting library receives as input the 3D data as observations, the distance function as objective function, and pointers to the gradient calculation functions, as well as the initial values of the various model parameters.

At each iteration, the hierarchy (or articulated structure) will be updated with the latest optimised parameter values,  and optimisation will proceed until an acceptable solution is found, in compliance with the set constraints, or until the maximum number of iterations has been reached (possibly without the optimal solution having been found).

The application WinSceneGraph interfaces the fitting library LibFitting and the articulated structure library LibFittingGraph. It allows the user to load a model from a text file, organised as a succession of nodes and their type, this being then organised into a Fitting Graph. The user can then load the 3D data, and launch the fitting of the model to this data, setting the number of iterations, and viewing the result as an OpenGL display. This display can be modified interactively by the user, namely the user can set the translational and rotational parameters of the model, view the update, and run a fitting process at any time.

The main fitting framework being in place, we can start working on the integration of biomechanical constraints to the fitting model, starting with the introduction of shoulder joint limits. So far, in the lab, the joints’ DOF were modelled as Euler angles, and the joint limits where thus simply minimal and maximal values of the rotations around each axis. The model was recently improved in the direction of joint boundaries represented as conic shapes with elliptic bases [3]. 

The idea we retained for representing these joint limits is using a consistent boundary space to represent the joint limits, and more specifically a quaternion space. The basics for this were laid out in the work of A. Hanson [2]. In a nutshell, the set of possible joint orientations and positions in space can be considered as a path of referentials in 3D space, where the rotation from one referential position to the next is calculated at each step. The sequence of quaternion rotations obtained can then be viewed as a path in quaternion space. In order to visualise this in 3D, we map these rotations to the S3 sphere, where all quaternion rotations are represented as points within this sphere. The line joining the centre of the sphere and a point represents the axis of rotation, and the distance from the point to the surface of the sphere contains the amount of rotation to be performed.

In our specific case of the shoulder joint limits, we used the lab's Vicon8 motion capture system for collecting 3D marker data corresponding to the entire range of possible shoulder movements, using markers on the lower and upper arm. For constructing the referentials, we use the upper arm segment as one axis, and then determine the normal to the plane represented by the lower arm markers. This yields two orthogonal axes, and the third one is then easily calculated to yield a referential for the current shoulder orientation and position. We perform the same calculation on each frame of the sequence, and thus obtain a path of referentials, as per above. The rotations from one to another are then mapped into S3, and we can see a few examples of such results in the below screenshots, depicting distinct flexion and twisting motions.

Around this cloud of 3D points, we can then wrap an implicit surface, thus yielding a volume. In order to determine whether we have exceeded joint boundaries, we need then only perform an inside/outside test on this volume. The algorithm for the reconstruction of an implicit surface by a cloud of points is based on the method outlined by Tsingos, Bittar and Gascuel for reconstructing medical organs [5]. Viewing of the results is performed re-using a marching cubes-based implicit surface visualisation tool of the lab.

[1] Grassia Sebastian F., "Practical Parameterization of Rotations Using the Exponential Map", Journal of Graphics Tools, vol.3, no.3, 29-48. 

[2] Hanson A.J., "Constrained Optimal Framings of Curves and Surfaces using Quaternions Gauss Maps", Symposium on Volume Visualization, Proceedings of the conference of Visualization '98 , October 18 - 23, 1998, Research Triangle Pk United States, p.375-382.

[3] Maurel W., "3D Modeling of the Human Upper Limb including the Biomechanics of Joints, Muscles and Soft Tissues", Ph.D. Thesis 1906 - Laboratoire d'Infographie - Ecole Polytechnique Federale de Lausanne – 1998. 

[4] Baerlocher P., Boulic R., Technical Report, March 1999, Plänkers. R, “On Jacobian Computation for Articulated Objects”, internal document.

[5] Tsingos, Bittar and Gascuel, “Automatic Reconstruction of Unstructured 3D Data: Combining Medial Axis and Implicit Surfaces”, Eurographics 1995, Tsingos, Bittar and Gascuel, “Implicit Surfaces for Semi-Automatic Medical Organs Reconstruction”, Computer Graphics International '95.

3. Planification de la suite des travaux (donner un calendrier approximatif)

4. Nouvelles publications (non mentionnées dans le rapport précédent)

L.Herda, P.Fua, R.Plänkers, R.Boulic, D.Thalmann, Skeleton-Based Motion Capture for Robust Reconstruction of Human Motion, Computer Animation '2000 Proceedings, Philadelphia PA, U.S.A., May 2000.

P.Fua, L.Herda, R.Plänkers, R.Boulic, Full Body Modeling Using Animation Models, ISPRS vol. XXXIII, Amsterdam, Netherlands 2000.