Code Update

I have recently added some modifcations to the avatar inside the code. William has added some code also into the face tracker application (specifically so that the window is ontop). This was needed so that the Virtual webcam will be able to capture the window via WinAPI GetClientRect, do display it on webcam programs such as skype. The Virtual Webcam code has been uploaded to the svn repository.

2D caricature Animation using the kinect

Recently i was informed about the new changes to the Kinect SDK and have seen that they have tutorials that show facial feature detection. I have used these to create a simple (using the kinect sample as a base) program which creates a 2d caricature that animates according to the users face. Below is a picture of me using this. I am also trying to add a 3rd screen that will create a 3d mesh that moves much in the same way as the 2d caricature. (This code has been added to the svn)

Update: faceoverlay

This is a new library that i have found that exists in the gst-plugins-bad set of plugins that uses OpenCV along with gstreamer to locate the face and overlay an image. At the moment i have not got this completely integrated because the library needs a dependencies which i have not been able to locate

Update: v4l2loopback

This is a loopback device that can act as a webcam for any program that can access a webcam. Essentially it takes in an input and that same input is its output for a program like Skype or Windows messenger. This essentially means that i can modify the input from a "real" camera with an overlay and transmit that to the v4l2loopback device which then Skype uses.

If i have time after the first prototype, i will try to integrate this with my prototype and possible show a live demonstration.

Links

http://code.google.com/p/v4l2loopback/

Lack of Blog Updates

I apologise for the lack of updates on this blog. Essentially i am in a coding phase. So at the present time i am just coding and am trying to put forward a prototype to show at the next meeting.

There is an update on my environment however. I have changed my coding environment from windows to linux for the sole reason that it is easier to install and use libraries.

Issues in Windows

Currently i am having some issues coding with GStreamer in Windows. It seems that i get a few errors that relate to creating a pipeline for the program to work with. Not sure if it is my environment or some other issue. So currently getting a linux distribution working on a computer for easier coding environment setup and should have a prototype done in about 1 week or 2.

If anyone viewing this blog has any experience with GStreamer on windows (www.gstreamer.com - sdk), and has any information for me it would be really appreciated.

For a little more information about the prototype. If i get the time for a UI then it will be done with the QT Framework

Current Modifications and What im Doing

Over these past few weeks, i have decided on a few changes to my system.

UI

The previous UI that i had prepared, now seems to be a little too "clunky". So i have decided to mainstream the UI to make it easier to use. Essentially there will be no video screen on the actual program, but instead will have options that the user will be able to set. I decided on this because this program, put simply is changing a webcam stream and replacing the face with a 2d caricature or 3d avatar, and because of this the UI does not need any screens of its own. The settings for the user to modify will include the choice of type of avatar (2d or 3d), then the avatar itself, along with a checkbox with the name activate which is checked when the apply button is pressed which then activates the program overlay, and others with if i find that more options would be useful.

System Structure

The system itself will be :

• GStreamer on the frontend that the UI code deals with
• OpenCV on the backend which does the algorithmic functions for face detection etc, called only via GStreamer functions made.

Essentially, the UI will call the GStreamer functions when the checkbox for activate (explained in the above paragraph) is on when the user clicks apply. At the moment i am trying to find the best method to pass the screen to OpenCV so that the OpenCV libraries can do the algorithmic functions. 

My current view is that i send via GStreamer a mat (or the relevant format) to OpenCV, which then calls functions for face detection and feature detection. OpenCV will then return a 2 dimensional array which are the coordinates of the features on the face, in reference to the mat that was sent to OpenCV. Then GStreamer will draw on the webcam video an overlay with the drawing according to the coordinates given by OpenCV.