Dimensionality Reduction
Principal Component Analysis
Find the orthogonal axes of maximum variance and project data onto the top k of them — compressing dimensions while retaining structure.
High-dimensional data has redundancy — correlated features compress. When features move together, the effective dimensionality is lower than the number of features. PCA finds those directions.
The columns of are the top eigenvectors of the sample covariance matrix, sorted by descending eigenvalue. Each eigenvalue is proportional to the variance explained by its component. Projecting a centered point onto gives a -dimensional representation that minimises reconstruction error.
3D view: Three clusters in 3D space with principal component axes (red = PC1, blue = PC2, green = PC3). Drag to orbit and see how the axes align with the directions of maximum spread.
▶sklearn — PCA
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
X_reduced = pca.fit_transform(X)
print('Explained variance:', pca.explained_variance_ratio_)Linear Discriminant Analysis
A supervised alternative to PCA that finds projections maximising class separation — the ratio of between-class to within-class scatter.
PCA maximises variance — it ignores class labels. The direction of most spread may have nothing to do with the direction that best separates classes. LDA takes a different approach.
is the between-class scatter matrix (weighted sum of squared distances from class means to the global mean) and is the within-class scatter (sum of covariances within each class). The optimal projection is the leading eigenvector of . With classes, there are at most discriminant directions.
▶sklearn — LinearDiscriminantAnalysis
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
lda = LinearDiscriminantAnalysis(n_components=2)
X_lda = lda.fit_transform(X, y)t-SNE
A nonlinear method that preserves local neighborhood structure — excellent for revealing cluster geometry in 2D, but not suitable for preprocessing.
t-SNE encodes high-D neighborhoods as probabilities. For each point, it computes a Gaussian distribution over neighbors. Points with similar high-D neighborhoods get high joint probability.
is the low-dimensional similarity between points and , modelled by a Student-t distribution with 1 degree of freedom. The gradient descent minimises the KL divergence between the high-D joint probabilities and . Perplexity controls the effective number of neighbors each point considers.
▶sklearn — TSNE
from sklearn.manifold import TSNE
tsne = TSNE(n_components=2, perplexity=30, random_state=42)
X_tsne = tsne.fit_transform(X)
# Note: no transform() — must retrain for new pointsChoosing a Method
PCA, LDA, and t-SNE answer different questions — matching the method to the goal matters more than any single algorithm.
Choose PCA for compression and preprocessing. PCA is fast, deterministic, and invertible. Use it before fitting models to reduce noise, speed up training, or visualise high-D data.
| PCA | LDA | t-SNE | |
|---|---|---|---|
| Type | Linear | Linear | Nonlinear |
| Supervision | Unsupervised | Supervised | Unsupervised |
| Structure | Global variance | Class separation | Local neighborhoods |
| Use case | Compression, preprocessing | Classification pipeline | 2D/3D visualization |
▶sklearn — PCA / LDA / TSNE
from sklearn.decomposition import PCA
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.manifold import TSNE
X_pca = PCA(n_components=2).fit_transform(X)
X_lda = LinearDiscriminantAnalysis(n_components=2).fit_transform(X, y)
X_tsne = TSNE(n_components=2, random_state=42).fit_transform(X)