GStreamer

During the week i have been looking for ways to implement stream modification and came across a library called GStreamer. It contains functions that allow for direct manipulation of a webcam stream even if the stream is currently being used by a program such as skype, which means that it does not take over the webcam stream like OpenCV. However OpenCV will still most likely be used due to the fact that facial recognition still needs to apply for the filters for GStreamer to use.

http://gstreamer.freedesktop.org/

Recent Activities and Progression

I realise that that blog post is over due but for good reason.

Firstly my project has been changed from a Waterfall Development Project to a Agile Development Project. This change was done because of the way that the code has been made. Me and William are currently sharing code and splitting the parts appropriately. While i have done more research than coding i am currently doing more coding than research because of my research in finding how my design can be implemented.

In my research i have found that to do the my implementation of being a go between between the video conferencing program and the kinect (or webcam), i need to implement a filter system. The filter uses the Windows SDK DirectDraw methods to hook into the webcam stream and modify the stream on the fly. This is exactly how i wanted to do this project but have had quite a few speed bumps along the way. First of which was finding out how to modify a webcam stream, since OpenCV is unable to do this an alternate route had to be found. This is where i found the filter method, while very promising there are very few open source examples of this type of implementation, not to mention that installing the Windows SDK was a more than a little hassle but will not go into that in the blog.

Currently i am re-evaluating my design with the filter method and find out how to use the Direct Draw methods in the SDK to be able to change the webcam streams, and doing more coding.

Facial Recognition Implementation

With the previous posts on the 2 different ways of facial tracking there arises 3 different ways to implement facial tracking into this project

1. Using AAM Tracking for Facial Recognition
2. Using HAAR for Facial Recognition
3. Using Both AAM and HAAR

The third option arose from my research on my discovery of this paper "Fast AAM Face Recognition with Combined Haar Classi ers and Skin Color Segmentation", written by various authors. It explains that since the AAM tracking is quite sensitive to the initial starting position of the model and image, it is possible to using HAAR classifiers to give the starting positions of the model and image, which would then filter to the AAM algorithm.

The implementation will be discussed with William Qi before any implementation is done due to how we are co-operating with the code.

Reference

http://www.jofcis.com/publishedpapers/2012_8_7_2799_2806.pdf

Active Appearance Model

Active Appearance Model (AAM) is a algorithm that uses a statistical model. This model is a model of the shape and grey-level appearance of an object. During the training phase of the algorithm, we begin to learn the relationship between model parameter displacements and the residual errors induced between a training image and a synthesised model. This algorithm is able to give a good overall match in just a few iterations even with poor starting estimates (to a certain degree).

However AAM is very sensitive to the initial matching position of the model and the image, and there could be problems with the computational expense of the algorithm and its accuracy without a good starting place.

References

T.F. Cootes, G.J. Edwards, C.J. Taylor. Active Appearance Models. 1998. Proc European Conference on Computer Vision.
(http://www.cs.cmu.edu/~efros/courses/AP06/Papers/cootes-eccv-98.pdf)

HAAR Object Detection for Facial Detection

This detection is also called "Viola-Jones object detection framework", named after Paul Viola and Michael Jones, which uses Haar Features (which derive from HAAR Wavelets) to detect objects. Haar-like features are features represented as digital images such as lines and edges that are used in object recognition. The Haar classifier uses these digital images to detect objects by viewing the change in contrast values between adjacent rectangular groups of pixels. These changes in contrast determine relative light and dark areas. The reason why these feature are used is because they are easily scaled by increasing or decreasing the size of the pixel group being analysed.


Figure 1. Haar Features

Using the Viola-Jones framework, the features that are used involves the sums of the image pixels within the rectangular areas. While there are other classifiers that use Haar such as the Haar Basis Function, the Viola-Jones uses more than one rectangular area making it more complex and therefore able to detect more facial features

This Viola-Jones framework is the method included with the current OpenCV libraries for facial detection.


References

Michael Jones, Paul Viola. Robust Real-time Object Detection. 2001. Second International Workshop on Statistical and Computational Theories of Vision - Modeling, Learning, Computing and Sampling.
(http://research.microsoft.com/en-us/um/people/viola/Pubs/Detect/violaJones_IJCV.pdf)

Dr John Fernandez, Phillip Ian Wilson. Facial Feature Detection Using HAAR Classifiers. 2006. JCSC 21, 4.

Week 6/7

Design



Since my previous post there have been no current changes to the design of the main system. There has been one main change and that is instead of using openGL libraries such as Glut or FreeGlut, me and my associate student have concluded that it would be better to keep it as a directX program since our platform for this thesis  will just be Windows.

Coding

The main part of these 2 weeks has been coding.

The first part was coding the kinect interface properly to receive data from the kinect. However this proved more challenging than originally expected and with the help of William and Hamed (a PhD Student/Researcher at the university), i have more understanding on how it works.

The second part of the coding is for the face detector. This is where my research of these 2 weeks comes in (written below).

The coding for this thesis has been more challenging than originally expected.

Research

During these past 2 weeks my main bulk of research was to look into the face detectors. Since there are more than one algorithms that could be used i narrowed them down to 2 main ones:

1. HAAR Object Detection - this is a library in OpenCV that has a face detection algorithm built into it
2. AAM Tracking - This is an external library that has been built to integrate into OpenCV.

My research into this has shown me that AAM tracking is more refined and is able to detect the face with a better accuracy than HAAR. I have not completed all my research into this, and when i have a blog post will be done on just this.


Current Implementation

Currently im in the process of implementing the HAAR detection because it comes ready with OpenCV. If my research shows me that AAM is a better algorithm in all aspects, steps will be taken to integrate AAM Tracking.

Week 5

For convenience all blog updates will be done on a weekly basis on the friday's of each week. Each blog update will be split up into 3 sections. Design, where i update on any design changes that may have occurred during the week. Coding, where i update on any code that has been done or is in progress over the week. And literature, where i add updates on any papers that i have read and deem have been useful in this specific context.

Design

As of the meeting that was held on the 4th of April 2012 these are what my designs were

Data Flow Diagram

HIPO Diagram

UI

The design has not been modified since said meeting.

Coding

During my constant communication with my colleague William Qi (http://anonskypewilliam.blogspot.com.au/?view=classic), we have come to the conclusion of using the Kinect SDK with OpenCV to be our coding base with the Kinect for such things as collecting data, face tracking etc.

We have also concluded that as a base for our thesis we would like to have 2D Deformation done, due to the fact that 3D Deformation could be quite CPU and Graphically intensive and possibly unviable until further research is done.

My current coding is in progress and it involves converting the input code from DirectX to OpenGL (OpenCV) in the Skeletal Viewer that is included in the Kinect SDK.

 Current Literature

All the current Literature that i have read and analysed include

1. Automatic reconstruction of personalized avatars from 3D face scans by Michael Zollhofer, Michael Martinek, Gunther Greiner, Marc Stamminger and Jochen Sußmuth
2. Realtime performance-based facial animation by Thibaut Weise, Sofien Bouaziz, Hao Li and Mark Pauly
3. Computer-Based Analysis of Facial Action in Schizophrenic and Depressed Patients by Frank Schneider, Hans Heimann, Waldemar Himer, Dietmar Huss, Regina Mattes, and Birgitta Adam

Am also currently reading several papers that include information about 3D expression accuracy  and kinect facial recognition algorithms.