Category Archives: Research assignment

Getting Kinected

The introduction of the Kinect Sensor was one of the main reason I finally decided to hop back onto the console train. My last console at the time was the Playstation 2 and having just upgraded my PC to play Crysis 2, there was no reason for me to purchase any of the next gen consoles (XBOX360 and PS3) at the time. That all changed when the Kinect was introduced. This new invention and coming off the buzz surrounding the Nintendo Wii console I felt was quite the game changer. A device to measure your body movement while being able to play HD games. This sold me and by the looks of things I was not the only one a bit interested, Microsoft gained the Guinness World Record for fastest selling consumer electronic product ever due to this little beauty.

Its only pitfall were that most of the games released did not diverge from the common gaming experience, ideas which never explored its full potential, a lot I feel was due to its secrecy with its API as well as not taking risks with how to use the device in games. Even the release of the official SDK was delayed and was only supplied from Microsoft after the community had successfully hacked the Kinect first.

Moving forwards to today I was always intrigued by the devices ability to capture motion mainly its skeletal tracking ability. This fascination stemmed from my other interest in being able to produce motion capture at a cheaper rate than the common and vastly expensive motion capture systems available at time of writing. Being a programmer, my animation skills was always non existent. I had delved into using 3dsMax back when it was version R12 after seeing someone use a bootleg copy of it for their final year project animation of a device from an AutoCAD schematic in 2001. I played around with it but unless I continued to use bootleg versions the cost was always a diminishing factor to take it in anyway seriously, so I never pursued it any further and stopped after a couple of years.

The idea instead remained in the back of my mind until this year when I was given opportunity as part of my studies at AIE to pursue a topic that I was interested in. Kinect motion capture came to mind immediately. After checking where the release of the SDK had come, 1.7 at time of writing I felt it was the right time to re-explore and take my ideas and make them realities. The community had indeed been active once I got myself a hold of version 1.7, lots of communities had formed all doing cool and interesting things. Now all I needed to do was to come up with a way of taking the motion capture to the next level and create something to record it. This is where Unity comes in to fill the gaps. I figured there was enough available through examples in the SDK and what I had become familar with in Unity to allow me to capture that motion easily, all that was needed was for me to write up the code to record that animation so it could be captured, recorded and played back all while in Unity. Having familiarise with Unity alot over the last few weeks and being introduced to it last year everything about it seemed plausible, so this is where my research assignment begins.


Research assignment – Kinect sensor

As part of this years assessment we are given the opportunity to research an advanced topic on anything we so desire within the games industry. Provided it wasnt something we had not previously researched prior and wasnt something we could produce over the weekend or last minute. 104 hours is the recommended time one needs to spend on research and it is to cover the entire year, with presentation at the end in front of peers and academia.

So with that I have chosen a subject that I have had some interest in, the kinect sensor. Mainly the areas of animation and motion capture, it also has speech recognition, a hand gesture API and depth sensing and various other items that I wont list here. Granted if I have time I will visit each of the topics individually, I have decided I want the largest time allocated for motion capture and animation.

Here is an outline of the milestone and topics I want to visit and present.

Human skeleton and joint movement

  • Tracking hand movements and skeleton tracking IDs
  • Processing skeleton frames
  • Bones and joints tracking and live feedback
  • Motion capture and controlling rigs

Colour image data stream

  • Event and polling
  • Enabling, disabling, attaching processing and rendering

Depth image processing

  • Triangulation
  • Depth frame
  • Calculating distance and colourizing data
  • 3d view of depth, creating model

Speech recognition

  • How it works
  • Speech recogniser

Human gesture recognition

  • Gesture recogniser
  • Algorithmic gesture – extending gesturehelper class to include zoom in, zoom out, swipe to right classes.
  • Neural networks

Kinect in the cloud

  • Windows azure
  • Remoting using windows phone

Depending on time and how difficlt each topic might be these may or may not change in future.