Difference between revisions of "Android Head Tracking"

Revision as of 03:12, 7 June 2013

Project Overview

This project's main goal is to be able to create an application on the Android mobile platform that will use face detection to track the head movements of the user. The face detection is used with OpenCV and Androids Face Detection API, and with these algorithms, we are able to locate the face with the device's front facing camera. After face detection has been implemented, we would need to be able to track the faces X,Y and Z coordinates (relative to the camera). With this data, we would be able to create a 3D model view where the user would be able to look around the object by simply moving their head.

Significant Updates

• June 3: Generated formula for distance of face relative to center of screen.
• May 29: Separated the core parts of the program to allow the software to be added into any android app.
  • 2 src files
  • 3 raw files
  • OpenCv Libraries
• May 23: Able to hide the preview completely, or partially (based upon user preference)
• May 20: Removed most of the bloat from sample code and was able to create a new android app with core files.
• May 16: Got middle of eye position and was able to give a live display of values on view
• May 8-9: Discovered helpful Face Detection/Eye Tracking algorithm, and successfully set up environment and code to work as a base.
  • Android Eye Detection and Tracking with OpenCV
• May 7: - Got face detection samples running on android device (complied on development computer)
• April 30: - Found Error with Hardware on API on Development device.
• April 23: - Gained Control of the Camera

TO DO

• Using openCV for face detection
  • Testing face detection samples on development device.
  • Finding the range of Face Detection
• Display eye positions, and other helpful information
• Enable Front Camera
• Comb through Eye Detection code to create usable functions
• Centralize the coordinates to have the middle of the screen to be (0,0)
  • Figure out a way to do Z coordinate
• Remove the preview and still be able to output face coordinates.
• Create simple openGL interface

Known Bugs/Issues

• Android Face Detection API
  • Only works for Ice Cream Sandwich [API 14.0 4.0] and up
  • Not all Devices Support the FaceDetection API
• OpenCV Samples
  • Sample Face Detection Code is not the most accurate face detection
    • Need to do more testing on faces and different lighting and back ground
  • Face Detection works when user looks directly at the camera but has issues finding sides of the face.
• Eye Tracking
  • Pupal are not tracked correctly like in the tutorial but I will not being using that data for this project and will focus on bounding box around the eyes for coordinates
• The range of face detection is still unknown but at the same time, it is not very far.
• There are multiple issues with false positives that might cause some issues with the code.
  • When multiple faces are detected, I don't know how to keep track of the main face we are looking at.
  • Faces are kept in a array of faces
• The coordinates that are obtained are based upon the size of the screen of the device.

Installation

Files

• All files needed will be uploaded to GitHub

Setting up Environment

Other than the normal Android Environment; the OpenCV Android environment must also be installed onto your workstation.

• Documentation for installation are here
  • My personal environment was set up on a Mac with the Eclipse IDE.
• OpenCV does have samples that you can run. Try to run the Face Detection sample to make sure all of the libraries have been installed correctly.

Loading Src Files

• There are 2 core files that needs to be added into your src directory and 1 Optional
  • Core
    • FdView.java
      • Where all of the analysis for face detection is done
    • SampleCvViewBase.java
      • The interface for the surface to display our preview.
  • Optional
    • FpsMeter.java
      • Not absolutely needed for Face Detection

Additions to Activity

• First this function needs to be added into your main activity or where you would want to start to load the Open CV Library. Within this function, we are creating initializing the view and also loading the view into our layout.
• The function requires a layout that is being used by the application and also a width and hight of how large you want the preview to be.

public BaseLoaderCallback setup(RelativeLayout frame,int width, int height){
	Display display = getWindowManager().getDefaultDisplay();
	Point size = new Point();
	display.getSize(size);
	final int display_width = size.x;
	final int display_height = size.y;
	final int w = width;
	final int h = height;
	final RelativeLayout frameLayout = frame;
	  BaseLoaderCallback output = new BaseLoaderCallback(this) {
		 @Override
			public void onManagerConnected(int status) {
				switch (status) {
				case LoaderCallbackInterface.SUCCESS: {
					Log.i(TAG, "OpenCV loaded successfully");

					// Load native libs after OpenCV initialization
					// System.loadLibrary("detection_based_tracker");
						
						
					// Create and set View
					mView = new FdView(mAppContext,frameLayout,display_width,display_height);
					mView.setDetectorType(mDetectorType);
					mView.setMinFaceSize(0.2f);

					
											
					frameLayout.addView(mView, new ViewGroup.LayoutParams(w, h)); // Adding to specific size
					//frameLayout.addView(mView,0); // Adding over all View

					setContentView(frameLayout); // Setting the Frame Layout as what we see. 

					// Check native OpenCV camera
					if (!mView.openCamera()) {
						AlertDialog ad = new AlertDialog.Builder(mAppContext)
								.create();
						ad.setCancelable(false); // This blocks the 'BACK' button
						ad.setMessage("Fatal error: can't open camera!");
						ad.setButton("OK", new DialogInterface.OnClickListener() {
							public void onClick(DialogInterface dialog, int which) {
								dialog.dismiss();
								finish();
							}
						});
						ad.show();
					}
				}
					break;
				default: {
					super.onManagerConnected(status);
				}
					break;
				}
			}
	    };
	    
		    return output;
    
    }

• This part of code needs to be added into your onCreate or your onResume since it will load your OpenCV libraries into your application so the rest of the program can use.

 
// Calling the setup to create a BaseLoaderCallBack for OpenCv
// The function code for this is above
BaseLoaderCallback mOpenCVCallBack = setup(new RelativeLayout(
				getApplicationContext()),width,height);
		

Log.i(TAG, "Trying to load OpenCV library");
if (!OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_2_4_2, this,mOpenCVCallBack)) {
	Log.e(TAG, "Cannot connect to OpenCV Manager");
}
 

Loading RAW files

• There are 3 raw files that needs to be added to the res directory, and placed into the raw folder.
  • if there is not a raw folder, then create one and add the files
• Files
  • "haarcascade_lefteye_2splits.xml"
  • "haarcascade_righteye_2splits.xml"
  • "lbpcascade_frontalface.xml"

Permissions

    <uses-permission android:name="android.permission.CAMERA"/>
    <uses-feature android:name="android.hardware.camera" />
    <uses-feature android:name="android.hardware.camera.autofocus" />

Results

Detection of Face with Coordinates [Headtracking.apk]

Demo [Headtrackingdemo.apk]

Face Detection Methods

Android API

• The following code allows determines if the hardware of the device support Face Detection API

// if faces is less than zero then device does not support face detection 
int faces = params.getMaxNumDetectedFaces(); 

• Used within a check function

/** 
     * faceDetectionCheck Method
     * @param currentCamera	the current Camera used for the application
     * Checks to see if the current camera passed 
     * @return
     */
    private static boolean faceDetectionCheck(Camera currentCamera){ 
    	Camera.Parameters params = currentCamera.getParameters();
        System.out.println("Checking to see if face detection works on the Device");
        // start face detection only *after* preview has started
        int faces = params.getMaxNumDetectedFaces();
        if(faces > 0){
        		System.out.println("Android Face Detection is compatable on this Device");
        		return true;
        }
        System.out.println("Android Face Detection API is not compatable");
    	
    	return false;
        }
    

OpenCV Samples

• This the the result from OpenCV face detection sample.
  • 

OpenCV Eye Tracking

• Base Tutorial 1 model running on Asus Transformer
  • Tutorial can be found at Android Eye Detection and Tracking with OpenCV
  • 

Additional Information

Project Goals

• Acuire Face Detection on Android Devices
• Calculate distance of face in millimeters
• Create a OpenGl app using head tracking

Future Uses

• The code can be used on any android app
• Allowing users to view 3d object
• Possibility to view 3d items online
• Head Tracking

Developers

Software Developer

• Ken Dang

Project Advisor

• Jurgen Schulze

Resources

• Android Developers
• Asus TF300T
• Eclipse (Mac)