Support Vector Classifier (SVC) is part of the Support Vector Machines (SVM) family, which are supervised learning models used for classification and regression analysis.
SVM includes various implementations such as SVC for classification and SVR for regression. The goal of SVM is to find a hyperplane that best separates the classes in the feature space. Support Vector Classification (SVC) is a binary classifier that aims to classify the data points into one of two classes.
Basically, a Support Vector Machine (SVM) is a supervised machine learning algorithm that classifies data by finding an optimal line or hyperplane that maximizes the distance between each class in an N-dimensional space [2].
For example, in 2-D space, hyperplanes are the lines.
L1 does not separate the classes. L2 does, but only by a small margin. L3 separates them by the maximal margin.
Classifying data is a common task in machine learning. Suppose some given data points each belong to one of two classes, and the goal is to decide which class a new data point will be in. In the case of support vector machines, a data point is viewed as a p - dimensional vector (a list of p numbers) and we want to know whether we can separate such points with a (p - 1) dimensional hyperplane. This is called a linear classifier. There are many hyperplanes that might classify the data. To extend to multi-class classification we can use strategies like one-vs-one or one-vs-all.
Support Vector Classification (SVC) is a versatile and powerful machine learning algorithm that's widely used in various applications such as text classification, image classification, Geographic information system (GIS), bioinformatics, financial analysis, and customer segmentation and so on. In this blog post, we will explore how to use the Support Vector Classifier (SVC) from the Sklearn library to classify images.
How to use it?
To get started with image classification using SVC in Sklearn, we'll need to install the required libraries. We can use pip to install them.
pip install numpy pandas scikit-learn
pip install opencv-python
1. Prepare dataset
We have to import the necessary libraries for python for getting start
import os
import cv2
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score, classification_report
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
2. Import library, load
We need to prepare both training set and testing test separately that there is no the same item of testing data in training data. For example we can download FER 2013 for researching.
Assume that we have a directory "dataset" with images categorized ("happy", "sad", "neutral") into folders based on their classes
def load_data(data_dir):
labels = []
images = []
label_map = {"happy": 0, "sad": 1, "neutral": 2}
print("Start load:" + data_dir)
for label in label_map.keys():
print(f'loading category:{label} successfully')
folder_path = os.path.join(data_dir, label)
for img_name in os.listdir(folder_path):
img_path = os.path.join(folder_path, img_name)
img = cv2.imread(img_path, cv2.IMREAD_GRAYSCALE)
img = cv2.resize(img, (48, 48))
images.append(img.flatten())
labels.append(label_map[label])
print("Finish")
return np.array(images), np.array(labels)
train_data_dir = "dataset/train"
X_train, y_train = load_data(train_data_dir)
test_data_dir = "dataset/test"
X_test, y_test = load_data(test_data_dir)
4. Use Principal Component Analysis (PCA) to reduce the number of dimensions (features) in dataset.
PCA allow to transforms a large set of variables into a smaller one that still contains most of the information in the large set.
pca = PCA(n_components=64)
X_train_pca = pca.fit_transform(X_train)
X_test_pca = pca.transform(X_test)
3. Train SVC model
We can create an instance of the SVC model and train it on the training data as follow:
svc_model = SVC(kernel='linear', C=0.01, random_state=40)
svc_model.fit(X_train_pca, y_train)
3. Make predictions
Use the trained model to make predictions on the testing data as follow:
y_pred = svc_model.predict(X_test_pca)
4. Evaluate the Model
We evaluate the performance of the model using accuracy and classification report
accuracy = accuracy_score(y_test, y_pred)
print(f'Accuracy: {accuracy * 100:.2f}%')
print('Classification Report:')
print(classification_report(y_test, y_pred))
The accuracy of model only 55.71%. So we need to fine-tune hyperparameters for improving it.
5. Fine-tune hyperparameters and retrain
- The parameters of SVC can be tuned to optimize SVC model about result of classification and performance. It help to find the best combination between parameters. We can use Grid Search to search through a specified subset of hyperparameters. The goal is to find the combination that yields the best model performance based on a specified evaluation metric.
from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
# Define the parameter for grid
param_grid = {
'C': [0.1, 1, 10, 100],
'gamma': [1, 0.1, 0.01, 0.001],
'kernel': ['rbf']
}
# Create a grid for searching
grid = GridSearchCV(SVC(), param_grid, refit=True, verbose=2, cv=5)
# Fit the model
grid.fit(X_train_pca, y_train)
print("Best Parameters:", grid.best_params_)
print("Best Score:", grid.best_score_)
After we choose the best parameters and retrain model with new one.
6. Visualization results
Now, we visualize some of the test images along with their predicted labels
label_map_reverse = {0: "happy", 1: "sad", 2: "neutral"}
fig, axes = plt.subplots(3, 3, figsize=(9, 9))
for i, ax in enumerate(axes.flat):
img = X_test[i].reshape(48, 48)
ax.imshow(img, cmap='gray')
ax.set_title(f'Pred: {label_map_reverse[y_pred[i]]}, True: {label_map_reverse[y_test[i]]}')
ax.axis('off')
plt.show()
Here is some of the test images
Conclusion
There are many application of SVM such as Image Classification, Text Classification, Fraud Detection, Recommender Systems, Bioinformatics, Geographic information system (GIS).
In this blog post, we've only walked through the process of using the SVC classifier from Sklearn to classify images. SVC is a powerful tool for many classification tasks, and it can achieve high accuracy with the right parameters and good dataset. It is highly effective in binary classification tasks and can be extended to multi-class classification using strategies like one-vs-one or one-vs-all. However, we need to prepare a good dataset and to find out a specified subset hyperparameters so that we can yields the best model performance.
Reference:
[1] https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html
[2] https://www.ibm.com/topics/support-vector-machine
[3] https://www.csie.ntu.edu.tw/~cjlin/papers/libsvm.pdf
[4] Help from Microsoft Copilot
[5] https://www.freepik.com/free-vector/futuristic-circle-portal-glowing-black-background_39204093.htm#fromView=image_search_similar&page=1&position=40&uuid=1011aee1-cb07-45a5-a128-9ee90690dc34