header photo

Level 3b - Live Multiple Face Detection

Make a Windows Form Application in C# using EmguCV library, such that
1- it gets an input image from a live web camera OR from some folder by loading/reading it.
2- detects faces (one or many) in the input image.
3- and shows these detected faces in the image by marked rectangle/ box .



In my earlier post Getting Ready For Face Detection , i had collected and shared most of the basic concepts behind face detection that you need to know before implementing face detection. If you have not read that post then I strongly suggest you read it before diving into the implementation details, to avoid any sort of confusions as you proceed with this tutorial. Because that post provides you with brief materials to read, about:

  1. Machine learning and its relation to face detection.
  2. What is face detection?What are the different approaches to face detection?
  3. Which approach is best of all?
  4. Our Selected approach: Viola Jones Algorithm.
  5. Reasons for selecting Viola-Jones method
  6. Concept of how Viola Jones algorithm works – The Haar Cascade

 Let me summarize for you:

  • In a nutshell, we will implement Viola-Jones algorithm because it’s the easiest ready to use face detection method which is supported by OpenCV/EmguCV and has great results.
  • In this method, OpenCV (and therefore EmguCV) uses a type of face detector called a Haar Cascade classifier – the one introduced by Viola and Jones.
  • The Haar Cascade is a classifier (detector) trained on thousands of human faces.
  • This training data is stored in an XML file, and is later used by the classifier during detection.

NOTE: 'Classifier' means  the same thing as 'detector'!
The image, on which detection must happen, could come from a local hard drive or directly from a camera.  When the Viola-Jones object(face) classifier/detector runs over the image, it uses the trained data stored in an XML file's sometime earlier. With help of this data, the classifier decides if the identified object is a the object in question or not. in case of face detection, it decides if its “Face” or “Not Face”.

 Viola Jones Classification assumes a fixed scale for the object(face in our case), say 50x50 pixels. Since the object(face) in an image might be smaller or larger than this, the classifier runs over the image several times, to search for objects(faces) across a range of scales. This may seem an enormous amount of processing, but thanks to algorithmic tricks, classification is very fast, even when it's applied at several scales. The parameters to the classifier in the code control and "tune" the classifier's properties of detection.
Question: "Does Viola-Jones Classification/Detection work in the same way for any object? be it a face, eye, car, ball etc etc?!

Indeed! Viola-Jones' Haar Classifier is a generic detector, whose classification depends on the trained data stored in the XML it uses. so whatever object was used to train this XML, the classifier will detect that object in the image.e.g, if XML holds the trained data for human faces, then Viola-Jones Haar Classifier becomes "face detector". but if XML had data stored about a car, then the classifier works as "Car detector".


Now, let’s begin with implementing Viola-Jones method of face detection, and as we do, we’ll gain a better understanding of how it all fits together: the Haar classifier + the trained XML file + the parameters that “tune” the classifier.    

 I'll be taking the Camera Capture application, that we had last modified in Level 2 tutorial,  to a more advanced level in this tutorial.
For any details on Camera Capture application, or to start a project from scratch, please see  Level 1 and then Level 2 to avoid any confusion or lack of information on the application during this tutorial.

 NOTE FOR THOSE WHO ARE implementing face detection on an image stored in some  folder: you DON'T need Level 1 or 2's Camera Capture application; Feel free to use this ready-to-use EmguCV project in VS2010 and add the face detection code to it as mentioned in this tutorial. I suggest you to code behind the start button.

Please watch the vimeo  video- it has all steps in single video BUT to get explanation of each step in video, please read the text of each step as given below this video

STEP 1: DECLARING THE CLASSIFIER - Declare an object of class HaarCascade
The Viola-Jones classifier/detector used in EmguCV for detecting faces in an image is called HaarCascade. Lets declare a global object of class HaarCascade and call it ‘haar’. We’re making it global to allow any and all functions in our code to use it:  


private HaarCascade haar;            

(Please stream all the youtube videos at 720p for best visibility of code.)

STEP 2: LOAD THE HaarCascade XML file

A classifier uses data stored in an XML file to decide how to classify each image location. So naturally, haar will need some XML file to load trained data from.
You'll need to tell the classifier (haar object in this case) where to find this data file you want it to use. It's better to locate the XML file we want to use and make sure our path to it is correct, before we code the rest of our face-detection program.
Question:Do I have to first train an XML file for face or object detection myself?!

Answer:  For face detection, NO! Not at ALL! Please do NOT be mistaken. You do NOT have to train an XML file for face detection yourself!! Thanks to Viola & Jones and other people who had taken on this strenuous job, they have already trained XMLs with thousands of face samples. So those XML files are bound to give good results, up to 100% as I’ve seen. All you need is to use these available trained XML files.  

However if it’s an object other than a ‘face’ that you’re trying to detect, say, a car, THEN yes! You may need to train an XML file ONLY IF you do not find any such XML over the internet. Download these XMLs & see if there's the one you were looking for.
I'll be using the XML file called haarcascade_frontalface_alt_tree.xml It had the best results, as far as I have tried, for frontal face detection. Your experience might differ depending on your requirements, so you may freely try other xml files available for face detection. You can Download Here the various XML files I had found for frontal face detection in OpenCV setup and EmguCV setup folders.

The haar object we declared earlier will hold the data loaded from this XML file. I prefer to load the XML file once and for all, when the windows form loads, so lets initialize our haar object in the Form_Load function (CameraCapture_Load() in our 'Live face detection' case) as follows:

// adjust path to find your XML file 
haar = new HaarCascade("haarcascade_frontalface_alt_tree.xml");

NOTE: If you're using ReadyEmguCVProject.sln as i provided above, please double click the 'ReadyEmguCVProject windows form' so that it creates and takes you to the load function.

Question:Why is there no path specified for XML file in your code? Where have you put this file?”

  That’s because I had copied the XML file to the Debug folder of this project and am using it from there in my code. This way you need not specify any path to a file that’s in your debug folder, you only have to specify the full file name with its extension.
This also removes file location dependency for your XML; now that it’s with your other debug files, it makes no difference where you move the original XML file or your Debug folder to. Your code will execute without trouble!


--Selecting 'where' face detection must happen in the code.

Face detection will naturally need an image to process. Here comes the question that “where and when does the face detection take place?” Should it be an image you saved earlier from somewhere, or should it be a live image from a web camera connected to your application? That all depends on you! And the choice you make is what decides “where” your face detection lines of code fit into the rest of the code.

Since we’ve decided here to detect faces from a live image stream, we’ll add the code inside our ProcessFrame() function, right after the command that fetches the frames from our web camera.  


ALTERNATIVE: You can instead read an image from some folder and apply face detection on it.   
You’ll have to load your image into ImageFrame instead of capturing an image from a web cam. For that, instead of code that fetches captured image, use the following lines of code : 

//Read an image from hard disk at location e.g ‘E:\IMAGES\’
Image InputImg = Image.FromFile(@"E:\IMAGES\MyPic.jpg"); 
Image<Bgr,byte> ImageFrame = new Image<Bgr,byte>(new Bitmap(InputImg));

And then insert the face detection code after  you have set this image source.
NOTE: the @ sign in the image path above code makes path valid without having to write double slashes in path. Remove @ sign and see the error.

After setting the source of image (where ours is live image captured by web camera),
insert the code that performs face detection functions:

COPY-PASTE THIS CODE(as given in video):
 FOR LIVE FACE DETECTION: Copy-Paste This code into the ProcessFrame()  function of Camera Capture Application.OR in the corresponding place in your own application.
 FOR FACE DETECTION ON LOADED IMAGE: Copy-Paste this code behind the 'Start' button of  ReadyEmguCVProject Application. OR in the corresponding place in your own application.

Let us understand step-wise and in detail what this code is doing:

4.1. Convert frame (Image) captured by Web camera to Grey-Scale:
After loading/reading the input image, it must be converted to gray-scale so the detector can work on it. the following command converts a color EmguCV Image (ImageFrame) to a gray-scale EmguCV Image (grayFrame)

//convert the image to gray scale
Image<Gray, byte> grayframe = ImageFrame.Convert<Gray, byte>();

4.2. Call the Detector -Detect faces from the Grey-Scale image

After converting the image to gray-scale, the following command is the call to run the face detector on this gray-scale image (called grayframe here).

//detect faces from the gray-scale image and store into an array of type 'var',i.e 'MCvAvgComp[] var faces =  grayframe.DetectHaarCascade(haar, 1.4, 4,                                         HAAR_DETECTION_TYPE.DO_CANNY_PRUNING,                                         new Size(25, 25))[0];

This function, DetectHaarCascade(), takes up to five parameters, the first being the classifier you’re using (‘haar’ for HaarCascade in our case), rest of the four parameters are used for “Tuning" the detector. And by tuning I mean, to detect a large face? small face? Set “is not face” criterion to a high threshold? or Lower threshold? etc etc. the parameters in our code is sort of “default values”.

AFTER, completely implementing the face detection in this tutorial, I highly recommend you to check out "tuning" the face detector by simply changing the parameter values in the call to DetectHaarCascade(). and see the change in detection result yourself!
So that you have a better understanding of what they do to the detector.

To see how to change the values and what are the valid values? please read “Tuning the Detector by Setting its Parameters” which explains what each of these 4 parameters does and what valid values you can assign to them.
To count the total number of faces detected in one call is simple, after detecting faces in the image, just add this:

 int TotalFaces = faces.Length;   // you can use TotalFaces from  here on, i'll only display it:

 MessageBox.Show("Total faces detected: " + TotalFaces.ToString());

faces in our code IS the array after all which has all the detected faces, yes? all you need is its length.
Thanks to Wendy for asking this very useful question :)

4.3. Mark detected faces within colored rectangle
drawing a rectangle on the detected face is NOT the job of the classifier/detector! You’re going to have to do it yourself explicitly. In the above code, the following lines do this job:

//draw a green rectangle on each detected face in image
                foreach (var face in faces)
                    ImageFrame.Draw(face.rect, new Bgr(Color.Green), 3);

4.4. Display the image with marked Detect faces
Now that our faces have been detected and marked with rectangles, time to display the image with detections!

//show the image in the EmguCV ImageBox
CamImageBox.Image = ImageFrame;

ALTERNATIVE: To Display an EmguCV image in Windows Form Picture Box, use this code instead of the one in 4.4 above:

 //Show the image in Windows Form PictureBox called "pictureBox1"
pictureBox1.Image = ImageFrame.ToBitmap();


STEP 5: Add Missing 'Emgu.Cv.CvEnum'  directive before debugging.


Do NOT forget to add the 'Emgu.Cv.CvEnum' directive


STEP 6 : Debug the program!
STEP 7: Handle Error 2: missing .dlls

.DLL files are missing?! - ADD the new/more .dll files Required for face detection functioning!


STEP 8: Successful Debugging!

A COMMON PROBLEM is: "What are the BEST parameter values for my detection?"
OR  "Face detection result is not up to my expectations/needs!  How do I improve the detection results?!"

Please Read the solution here.

SOURCE CODE - Live face detection  (VS 2010)
NOTE:if you're a beginner to Face detection, then i HIGHLY stress you read and follow the whole tutorial to understand and learn this project.The source code is here only to HELP you!



Tutorial by: Mahvish

Go Back

@Mahvish: Thanks for your bug free tutorials!!
For ones encounter the nullPointer when choosing image from folders, check if you forgot the line
haar = new HaarCascade("....");

Your tutorial is really an awesome one ,I didn't see thing like this before but please can you focus on the implementation of viola algorithm itself .. Thanks .

@ Mahvish : your tutorials are very good and iam learning soo many things from this tutorial thanks a lot for this and can you please tell me when you are going to upload then next tutorials and how do i connect to the data base could you please upload coding to connect the database. thank you

Halo Mahvish thx for tuts , i m working on my FYP in this semester about measuring the height of ppl using camera. I only have some basic knowledge eg. RGB normal,grayscale etc. so your tuts really come in handy , and the papers too as refrence for my FYP. God bless you ^^

I did the "from image" code (not from live camera), and when I pressed the start button it showed me the image with a green square. that`s what it suppose to show? and if so, if I do the live camera code, can I make my own database for the faces that it will need to detect?

all the tutorials are very nice and very useful for my project work thank you.

I'm using emgucv 2.4 version i follow the steps but i get this warning 'Emgu.CV.Image.DetectHaarCascade(Emgu.CV.HaarCascade, double, int, Emgu.CV.CvEnum.HAAR_DETECTION_TYPE, System.Drawing.Size)' is obsolete: '"Use HaarCascade.Detect function instead. This function will be removed in the next release"'

Please help urgently!

I am having a NullRefrenceException error during runtime and the block of code it point to is below:

var faces =
grayframe.DetectHaarCascade(haar, 1.4, 4,
new Size(25, 25))[0];

Hi. i have followed your tutorials I just tried what you said. But it will when i load xml file in my form loading method i get error "The type initializer for 'Emgu.CV.CvInvoke' threw an exception." Please provide solution for that
thanks and regards
sathish R