Kinect UI for 3D Pacman

3D Pacman

Attempt at the classic arcade game Pacman in a 3D, immersive virtual environment. Player plays as a first-person view navigating the maze while under hot pursuit from the Ghost. Models modelled in external 3D modeling software and loaded into the code as a .obj file, along with textures.

Project is incomplete:

Collision detection still buggy. It uses a simple 2D line intersection collision algorithm to test collision against walls. The program saves the vertices of the maze obj file and applies the same transformations to them, rids their z-values to create a 2D line map. Player's navigation history is stored as a x,y location and the vector between the current direction and the future direction creates a line segment that is used to test against the line map; if collision returns true the player's movement is prohibited.

So far the ghost also simply comes towards the player's current location, also disregarding walls. Texturing on the ghost also has a slight bug (eyes dont show).

Kinect UI

The main research was aimed to be gathering Kinect data and being able to implement a gesture controlled, device free user interface. Data from the Kinect is retrieved and managed by use of the open source library libfreenect by OpenNI / OpenKinect, which in turn uses libusb-1.0.

Currently the program uses a simple algorithm:

Using only the depth data retrieved form the Kinect's depth camera, we cut off all depth according to a tested threshold, so that anything beyond that distance is disregarded. The distance should correspond to just in front of the person so that basically only a raised arm would be seen. The "image" gathered is binarized and using a simple component searching algorithm different components on the "image" is found. In general a raised arm is all it will see and that component is given control over the game. The image is scanned from left to right column by column so the right arm is given priority for control if say, more than 2 arms are raised. The centroids of the components are calculated and its deviation from the center dictates the speed and direction where the player would navigate.

There are obvious issues with this algorithm. It works best if the player stands at the center of the Kinect's field of vision, and would mess up with noise / other people in the field of play. It is not ideal, however the initial assumption was that the Kinect would be mounted overhead looking directly down at the head of the player. This way the depth cut off would make more sense, and noise would be less of an issue as we can select something like the closest / tallest point as the priority. Then a similar technique for finding the centroids (moment calculations in computer vision) can be applied to find the orientation of the component and dictate the direction of travel.

Data from the Kinect is retrieved and processed every frame on a separate thread. The simplicity of the algorithm and the fact that we only use and process depth data is to maintain a smooth performance.

In game updates: included a screen that shows the binarized images of the Kinect's vision.