Udacity Nanodegree Capstone — Dog Breed Classification
Project Overview / Problem Introduction
As part of this project, a Convolutional Neural networks (CNNs) is trained that processes real-world, user-supplied images to identify if it contains a dog or a human face and then predict the closest breed the dog or the human face resembles with.
Typically, CNNs are deep neural networks that require a massive amount of data and therefore compute/training time. In order to reach an acceptable level of accuracy, a pre-trained model could be leveraged to reduce the training time and at the same time, leverage the learned weights/parameters of this model. This process comes under the window of transfer learning.
Project Goal
The intent is to train a dog breed classifier algorithm that detects if the input image contains a dog face or a human face, predicts the breed in the case of a dog’s face, and gives the resembling dog breed in the case of a human face as the input.
The threshold to accept the model/algorithm would be to achieve at least 60% accuracy on the test set.
Strategy to solve the problem
In the jupyter notebook template that Udacity provides, the strategy for solving this problem is as follows:
- Step 0: Import Datasets
- Step 1: Detect Humans
- Step 2: Detect Dogs
- Step 3: Create a CNN to Classify Dog Breeds (from Scratch)
- Step 4: Use a CNN to Classify Dog Breeds (using Transfer Learning)
- Step 5: Create a CNN to Classify Dog Breeds (using Transfer Learning)
- Step 6: Write your Algorithm
- Step 7: Test Your Algorithm
Before we deep dive into each of the steps, let’s quickly take a look at the metrics to evaluate the algorithm.
Metrics
The metric used to evaluate the results, in this case, is the “accuracy” metric.
Here is a quick description of how accuracy is computed in machine learning: the trained model is used to make the predictions, and the predictions are matched with the ground truth (correct labels), the fraction of images/instances that are correctly identified by the model is termed as the accuracy.
Exploratory Data Analysis
The Udacity team has provided ~8300 labeled dog images where the label is one of the 133 possible dog breeds. The data is split into the train, test, and validation sets.
As this is a classification problem, it would make sense to see the distribution of the 133 classes/breeds
The snippet above tells that the dataset is not perfectly balanced, but the good thing is that even the lowest represented breed has at least a decent number (greater than 25) of training examples
Implementation
Step 0: Import the Dog dataset
The code snippet above imports a dataset of dog images. We use the “load_files” function of the scikit-learn library to populate a few variables:
train_files,valid_files,test_files- NumPy arrays containing file paths to imagestrain_targets,valid_targets,test_targets- NumPy arrays containing one hot-encoded classification labelsdog_names- list of string-valued dog breed names for translating labels
Step 1: Detect Humans
This code cell demonstrates how to use this detector (pre-trained detectors from open cv) to find human faces in a given image
In order to use any of the face detectors, the images must first be converted to grayscale. The detectMultiScale function executes the classifier stored in face_cascade and takes the grayscale image as a parameter.
In the above code, faces is a NumPy array of detected faces, where each row corresponds to a detected face.
Each detected face is a 1D array with four entries that specifies the bounding box of the detected face.
The first two entries in the array (extracted in the above code as x and y) specify the horizontal and vertical positions of the top left corner of the bounding box. The last two entries in the array (extracted here as w and h) specify the width and height of the box.
Step 2: Detect Dogs
When using TensorFlow as the backend, Keras CNNs require a 4D array (which we’ll also refer to as a 4D tensor) as input, with the following shape:
(nb_samples,rows,columns,channels)
where
nb_samples corresponds to the total number of images (or samples), and rows, columns, and channels correspond to the number of rows, columns, and channels for each image, respectively
The path_to_tensor the function in the snippet above takes a string-valued file path to a color image as input and returns a 4D tensor suitable for supplying to a Keras CNN.
- It first loads the image and resizes it to a square image that is 224×224 pixels.
- The image is then converted to an array, which is then resized to a 4D tensor.
- In this case, since we are dealing with color images, each image has three channels (RGB)
- Likewise, since we are processing a single image (or sample), the returned tensor will always have a shape: “(1,224,224,3)”
The paths_to_tensor the function takes a NumPy array of string-valued image paths as input and returns a 4D tensor with the shape:
(nb_samples,224,224,3)
Here, nb_samples is the number of samples, or the number of images, in the supplied array of image paths.
Think of nb_samples as the number of 3D tensors (where each 3D tensor corresponds to a different image) in the dataset.
Step 3: Create a CNN to classify Dog breeds
Now that there are functions for detecting humans and dogs in images, we need a way to predict breeds from images. In this step, images are pre-processed and a CNN architecture is defined to be trained on the dataset
The intent here would be to just get at least 1% accuracy on the test set, as there are a possible 133 breeds/categories in the Image Net dataset and a random guess will provide a correct answer roughly 1 in 133 times, which corresponds to an accuracy of less than 1%.
Model architecture:
There are 3 convolutional layers and 3 layers of max-pooling in this architecture followed by an average pooling layer and a dense layer at the end
Kernel size of 2 is used and the “ReLU” activation is used in all of the convolutional layers
The dense layer at the end is used to create a fully connected neural network
Train the model
The above snippet enlists the details about the model training piece, model checkpointing is used to save the model that attains the best validation loss.
Load the model with the best validation loss
Test the model
Step 4: Use a CNN to Classify Dog Breeds (using Transfer Learning)
To reduce training time without sacrificing accuracy, the Udacity team has provided a snippet to show how to train a CNN using transfer learning.
In the following step, you will get a chance to use transfer learning to train your own CNN.
A VGG-16 trained model is used and on top of it, a global average pooling layer and a fully connected layer are added
The same process is repeated as in step 3 to train the model, store the model with the best validation loss, and it’s loaded again and tested on the test set
With the transfer learning and learned weights in place, and accuracy of ~39% has been achieved on the test set
Step 5: Create a CNN to Classify Dog Breeds (using Transfer Learning)
This is the same as the step except for the part that a different pre-trained model is used in this one, and in step 4, sample code was already provided by the team to demonstrate an example of how to work with the transfer learning
I’ve used the trained ResNet model for this step
The model is compiled, trained, the best model basis the validation loss is saved, and is loaded in memory and tested on the test set
Test accuracy of 81% has been achieved with this model
A function is then defined to use this trained model and make predictions on the new input:
Step 6: Write your own algorithm
Here the intent is to make the predictions for new input. The input is first passed through the dog detector (defined in step 2) to see if the input indeed contains a dog, if not, it is passed through the human face detector (defined in step 1) to check if it contains a human face
Post the dog/human identification, it’s breed or the closest breed (in the case of a human face as the input) is predicted
Step 7: Test your Algorithm
A few images are run through the algorithm to see its performance. Here are the snippets from the test runs:
Here the input image is a human face and it is correctly identified by the algorithm
Here again, the input image is a human face and it is correctly identified by the algorithm
For input image 3, the input is not either a dog or a human but it’s a cat, and the same is identified correctly by the algorithm
For input image 4, the input is a dog and the same is identified correctly by the algorithm
And for the input image 5, 6 — both the inputs contain a dog face which is correctly identified by the algorithm
Results
The following piece of information collates together the test set accuracy achieved using three models:
- CNN from scratch: ~1%
- CNN from VGG16 (using transfer learning): ~40%
- CNN from ResNet50 (using transfer learning): ~80%
The difference in the accuracy of the three models is because of the different architecture, and the difference in the total number of parameters in each approach. Typically, more number of parameters would increase the accuracy but up to a certain point beyond which it might then try to overlearn from the input and would take the direction of overfitting and then might take a toll on the accuracy as well.
In this case, with each advanced model, the accuracy has increased which helps us establish that the overfitting of the model is not the case here.
Reflection
Deep Learning has really accelerated the intelligence of the systems and is now penetrating almost every field.
In this project, the idea was to use a convolution neural network to train a classifier that would identify if the input image contains a dog or a human face and predicts the closest breed
Test accuracy of ~80% has been achieved leveraging transfer learning with ~60 seconds of training time and this could further be improved by augmenting the dataset, maybe trying even more complex architecture that would offer more number of parameters and allow to learn even complex patterns.
Suggested Improvements
The following techniques could be used to gauge the impact it creates on the model training, accuracy:
- Augmenting the dataset
- Experimenting with the Neural Network architecture: kernel size, number of layers, type of pooling, and so on