Support Vector Machine
Maximal Margin Classifier
When data is linearly separable, choose the hyperplane that maximises the distance to the nearest points of each class.
When classes are linearly separable, infinitely many boundaries exist. Any hyperplane that correctly classifies all points is technically valid — but which one to choose?
The margin width is — maximising it is equivalent to minimising . The constraint ensures all points are at least distance 1 from the boundary (in the normalised scale). Only the support vectors — the points at exactly — influence the solution.
▶sklearn — SVC
from sklearn.svm import SVC
# Hard margin (linearly separable only)
model = SVC(kernel='linear', C=1e6)
model.fit(X_train, y_train)
print('Support vectors:', model.support_vectors_.shape)Soft Margin SVM
Relax the hard margin constraint to handle overlapping or non-separable classes — the regularisation parameter C trades margin width against training error.
Real data is rarely linearly separable. The two-moons dataset has no line that perfectly separates the classes — we need a way to allow imperfect solutions.
The slack variable measures how far a point has violated the margin. A point is correctly classified with , on the margin at , and misclassified when . The hyperparameter C controls the penalty: large C = low tolerance for violations; small C = wide margin accepted.
▶sklearn — SVC soft margin
from sklearn.svm import SVC
# Soft margin — tune C
for C in [0.1, 1, 10, 100]:
model = SVC(kernel='linear', C=C)
model.fit(X_train, y_train)
print(f'C={C}: {model.score(X_test, y_test):.3f}')The Kernel Trick
Replace the inner product with a kernel function to implicitly operate in a very high-dimensional feature space — enabling non-linear decision boundaries at no extra cost.
Linear boundaries fail on non-linear data. A straight line through two-moons data gives a poor boundary — half the points will always be misclassified.
The RBF kernel computes similarity between points: 1 when identical, decaying exponentially as distance grows. The parameter controls the radius of influence — large means only very nearby points influence the boundary, producing tight, complex shapes. Small produces smoother, broader boundaries.
Common kernels: linear , polynomial , RBF (above), sigmoid. The choice of kernel encodes assumptions about the data geometry.
3D view: Inner circle (orange) and outer ring (blue) are not linearly separable in 2D. Lifting each point by x²+y² maps them to a space where a flat plane separates the classes. Drag to orbit.
▶sklearn — SVC
from sklearn.svm import SVC
# RBF kernel (most common default)
model = SVC(kernel='rbf', C=1.0, gamma='scale')
model.fit(X_train, y_train)
# Other kernels
poly_model = SVC(kernel='poly', degree=3, coef0=1)
sigmoid_model = SVC(kernel='sigmoid', gamma='scale')