Machine Learning Algorithms

Introduction to Machine Learning

Machine Learning is a subset of artificial intelligence that enables systems to learn and improve from experience without being explicitly programmed. ML algorithms build mathematical models based on training data to make predictions or decisions.

This chapter covers fundamental machine learning algorithms across different categories: supervised learning, unsupervised learning, and neural networks. Understanding these algorithms is essential for building intelligent systems, data analysis, and AI applications.

Machine Learning Categories

1. Supervised Learning

Algorithms learn from labeled training data to make predictions. The model is trained on input-output pairs and learns to map inputs to outputs.

• Classification: Predict discrete categories (e.g., spam/not spam)
• Regression: Predict continuous values (e.g., house prices)

2. Unsupervised Learning

Algorithms find patterns in data without labeled examples. The model discovers hidden structures in the data.

• Clustering: Group similar data points
• Dimensionality Reduction: Reduce number of features

3. Self-Supervised Learning

Labels are generated from the data itself — predict the next word, or reconstruct a masked patch of an image — so training can use unlabelled data at enormous scale. This is the paradigm behind every modern foundation model, and it is arguably the most consequential development in ML in the past decade. It sits between supervised and unsupervised: the training signal is supervised in form, but no human labelled anything.

4. Reinforcement Learning

An agent takes actions in an environment and learns from a reward signal rather than from labelled examples. The defining difficulties are delayed reward (an action's consequences may only appear much later) and the exploration-exploitation trade-off. Used in robotics, game playing, recommendation, and — as RLHF — in aligning large language models.

5. Neural Networks and Deep Learning

Layered function approximators trained by gradient descent. Not a separate learning paradigm so much as a flexible model family usable within any of the above.

Supervised Learning Algorithms

Linear Regression

Predicts a continuous target variable using a linear relationship between features and target. The algorithm finds the best-fit line through the data points.

# Simple linear regression: y = mx + b
def linear_regression(X, y):
    n = len(X)
    mean_x = sum(X) / n
    mean_y = sum(y) / n
    
    # Calculate slope (m)
    numerator = sum((X[i] - mean_x) * (y[i] - mean_y) for i in range(n))
    denominator = sum((X[i] - mean_x) ** 2 for i in range(n))
    m = numerator / denominator
    
    # Calculate intercept (b)
    b = mean_y - m * mean_x
    
    return m, b

# Predict function
def predict(x, m, b):
    return m * x + b
                

Logistic Regression

Used for binary classification. Uses the logistic function to map predictions to probabilities between 0 and 1.

import numpy as np

def sigmoid(z):
    """Sigmoid activation function"""
    return 1 / (1 + np.exp(-z))

def logistic_regression_predict(X, weights, bias):
    """Predict probabilities using logistic regression"""
    z = np.dot(X, weights) + bias
    return sigmoid(z)

def cost_function(y_true, y_pred, eps=1e-15):
    """Binary cross-entropy loss.

    Clip before taking the log: a confident-and-correct prediction can be exactly
    0.0 or 1.0 in floating point, and log(0) is -inf, which poisons the whole loss.
    """
    y_pred = np.clip(y_pred, eps, 1 - eps)
    m = len(y_true)
    return -(1/m) * np.sum(
        y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred)
    )

def train_logistic_regression(X, y, lr=0.1, epochs=1000, l2=0.0):
    """Fit by gradient descent. Without this, the two functions above are
    only half an algorithm - there was no way to obtain the weights."""
    n_samples, n_features = X.shape
    weights = np.zeros(n_features)
    bias = 0.0

    for _ in range(epochs):
        y_pred = sigmoid(X @ weights + bias)
        error = y_pred - y                       # gradient of BCE through sigmoid
        # The sigmoid derivative cancels against the cross-entropy derivative,
        # which is exactly why this pairing is used - the gradient is just the error.
        weights -= lr * (X.T @ error / n_samples + l2 * weights)
        bias    -= lr * (error.mean())

    return weights, bias
                

Logistic regression is often the right baseline: it is fast, it is interpretable (each coefficient is a log-odds contribution), and it produces reasonably calibrated probabilities out of the box, which many more powerful models do not. Always scale your features first — gradient descent converges slowly when features have wildly different ranges.

Decision Trees

A tree-like model that makes decisions by splitting data based on feature values. Each node represents a decision, and leaves represent outcomes.

The splitting criterion

The whole substance of a decision tree is how it chooses a split. The usual measure is Gini impurity: the probability that a randomly chosen sample would be misclassified if labelled at random according to the class distribution in the node.

import numpy as np

def gini(y):
    """0 = pure (one class only), higher = more mixed. Max 1 - 1/k for k classes."""
    if len(y) == 0:
        return 0.0
    _, counts = np.unique(y, return_counts=True)
    p = counts / len(y)
    return 1.0 - np.sum(p ** 2)

def entropy(y):
    """The alternative criterion. Very similar results; Gini is slightly cheaper."""
    if len(y) == 0:
        return 0.0
    _, counts = np.unique(y, return_counts=True)
    p = counts / len(y)
    return -np.sum(p * np.log2(p))

def find_best_split(X, y):
    """Find the (feature, threshold) that most reduces weighted impurity."""
    n_samples, n_features = X.shape
    if n_samples < 2:
        return None, None

    parent_impurity = gini(y)
    best_feature, best_threshold, best_impurity = None, None, parent_impurity

    for feature in range(n_features):
        for threshold in np.unique(X[:, feature]):
            left = X[:, feature] < threshold
            right = ~left
            if left.sum() == 0 or right.sum() == 0:
                continue                        # a split that separates nothing

            weighted = (left.sum()  * gini(y[left]) +
                        right.sum() * gini(y[right])) / n_samples

            if weighted < best_impurity:
                best_feature, best_threshold, best_impurity = feature, threshold, weighted

    # information gain = parent_impurity - best_impurity
    return best_feature, best_threshold
                

Building the tree

class DecisionNode:
    def __init__(self, feature=None, threshold=None, left=None, right=None, value=None):
        self.feature = feature
        self.threshold = threshold
        self.left = left
        self.right = right
        self.value = value  # For leaf nodes

def majority_class(y):
    values, counts = np.unique(y, return_counts=True)
    return values[np.argmax(counts)]        # np.unique, not y.count() - y is an ndarray

def build_decision_tree(X, y, max_depth=10, min_samples_split=2):
    # Stopping conditions: depth budget spent, node already pure, or too few samples
    if max_depth == 0 or len(np.unique(y)) == 1 or len(y) < min_samples_split:
        return DecisionNode(value=majority_class(y))

    best_feature, best_threshold = find_best_split(X, y)
    if best_feature is None:                # no split improves impurity
        return DecisionNode(value=majority_class(y))

    left_indices = X[:, best_feature] < best_threshold
    right_indices = ~left_indices

    left = build_decision_tree(X[left_indices], y[left_indices],
                               max_depth - 1, min_samples_split)
    right = build_decision_tree(X[right_indices], y[right_indices],
                                max_depth - 1, min_samples_split)

    return DecisionNode(best_feature, best_threshold, left, right)

def predict_one(node, x):
    while node.value is None:
        node = node.left if x[node.feature] < node.threshold else node.right
    return node.value

def predict(node, X):
    return np.array([predict_one(node, x) for x in X])

# Time: O(n_features * n_samples^2 * depth) for this straightforward version.
# Real implementations pre-sort each feature once and slide the threshold,
# reducing it to O(n_features * n_samples * log(n_samples) * depth).
                

Trees overfit badly if left unconstrained. A tree grown to full depth will memorise the training set exactly — every leaf a single sample, 100% training accuracy, and poor generalisation. Control this with max_depth, min_samples_split, min_samples_leaf, or by growing fully and then pruning back. In practice, this fragility is precisely why single trees are rarely used alone; see Ensemble Methods below.

K-Nearest Neighbors (KNN)

A simple, instance-based learning algorithm. Classifies data points based on the majority class of their k nearest neighbors.

import numpy as np
from collections import Counter

def euclidean_distance(point1, point2):
    """Calculate Euclidean distance"""
    return np.sqrt(np.sum((point1 - point2) ** 2))

def knn_predict(X_train, y_train, X_test, k=3):
    """K-Nearest Neighbors prediction"""
    predictions = []
    
    for test_point in X_test:
        # Calculate distances to all training points
        distances = [euclidean_distance(test_point, train_point) 
                     for train_point in X_train]
        
        # Get k nearest neighbors
        k_indices = np.argsort(distances)[:k]
        k_nearest_labels = [y_train[i] for i in k_indices]
        
        # Majority vote
        most_common = Counter(k_nearest_labels).most_common(1)[0][0]
        predictions.append(most_common)
    
    return predictions
                

Three things this simple version glosses over. Feature scaling is mandatory — Euclidean distance is dominated by whichever feature has the largest numeric range, so an unscaled "income" column in dollars will completely drown a "years of experience" column. Standardise before using KNN. The curse of dimensionality means that in high dimensions all pairwise distances converge toward each other, so "nearest" stops being meaningful past roughly 10–20 features. And this brute-force scan is O(n·d) per query: real systems use k-d trees or ball trees for low dimensions, and approximate nearest-neighbour indexes — HNSW, FAISS, ScaNN — for high-dimensional vector search, which is what every embedding-based retrieval system runs on today.

Support Vector Machines (SVM)

Finds the optimal hyperplane that separates classes with maximum margin. Effective for both linear and non-linear classification.

from sklearn import svm

# Linear SVM
def train_svm(X_train, y_train, kernel='linear'):
    """Train Support Vector Machine"""
    model = svm.SVC(kernel=kernel)
    model.fit(X_train, y_train)
    return model

# The algorithm finds the hyperplane that maximizes the margin
# between classes, using support vectors (data points closest to the boundary)
                

Ensemble Methods

A single decision tree is unstable: change a few training rows and you get a visibly different tree. Ensembles turn that weakness into a strength by combining many trees. This section matters disproportionately — for tabular data, gradient-boosted trees remain the strongest general-purpose method available, ahead of neural networks, and have been for over a decade.

Bagging and Random Forests

Bagging (bootstrap aggregating) trains each model on a random sample of the data drawn with replacement, then averages the predictions. Averaging cancels the individual models' variance without raising their bias.

A random forest adds a second source of randomness: at every split, each tree may only consider a random subset of the features (typically √p of them for classification). This decorrelates the trees — without it, one dominant feature would appear at the top of nearly every tree and the ensemble would barely differ from a single one.

from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(
    n_estimators=500,        # more trees never overfits - it only costs time
    max_features='sqrt',     # the decorrelation knob
    min_samples_leaf=1,
    n_jobs=-1,               # trees are independent, so this parallelises perfectly
    random_state=0,
)
model.fit(X_train, y_train)

# Out-of-bag samples (rows a given tree never saw) give a free validation estimate
# without holding data out - set oob_score=True.

Random forests are the best "just works" model: little tuning, hard to overfit by adding trees, robust to outliers, and no feature scaling required.

Boosting

Boosting is sequential rather than parallel. Each new tree is fitted to the residual errors of the ensemble so far, so every model corrects its predecessors' mistakes. In gradient boosting specifically, each tree is fitted to the negative gradient of the loss — which is what lets the same algorithm optimise any differentiable objective, not just squared error.

# Toy gradient boosting for regression - the whole idea in ten lines
predictions = np.full(len(y), y.mean())      # start from a constant
trees = []

for _ in range(n_rounds):
    residuals = y - predictions              # what the ensemble still gets wrong
    tree = fit_shallow_tree(X, residuals)    # depth 3-8 - a WEAK learner
    predictions += learning_rate * tree.predict(X)   # small steps
    trees.append(tree)

# Predict: start from the mean, add each tree's scaled contribution.

The two critical hyperparameters pull against each other. A small learning rate (0.01–0.1) means each tree contributes little, which generalises better but needs more rounds. Number of rounds is what actually overfits — unlike a random forest, adding boosting rounds indefinitely will degrade test performance. Use early stopping on a validation set.

The Three Implementations You Will Actually Use

import lightgbm as lgb

model = lgb.LGBMClassifier(
    n_estimators=2000,          # set high and let early stopping choose
    learning_rate=0.03,
    num_leaves=31,
    subsample=0.8,              # row sampling  - adds regularisation
    colsample_bytree=0.8,       # column sampling
    random_state=0,
)
model.fit(
    X_train, y_train,
    eval_set=[(X_valid, y_valid)],
    eval_metric='auc',
    callbacks=[lgb.early_stopping(100)],   # stop when validation stops improving
)

Bagging vs Boosting

Two further ensemble ideas worth knowing: stacking trains a meta-model on the out-of-fold predictions of several base models, and simple averaging of diverse models is often nearly as good for a fraction of the complexity.

A caution on interpretation: tree-based feature importances computed from impurity are biased toward high-cardinality features. Use permutation importance or SHAP values instead when the explanation actually matters.

Unsupervised Learning Algorithms

K-Means Clustering

Partitions data into k clusters by minimizing the sum of squared distances between data points and cluster centroids.

import numpy as np

def kmeans_plusplus_init(X, k, rng):
    """k-means++ (Arthur & Vassilvitskii, 2007): seed centroids far apart.

    Purely random initialisation frequently produces bad clusterings and empty
    clusters. k-means++ picks each new centroid with probability proportional to
    its squared distance from the nearest existing one, which gives an O(log k)
    approximation guarantee. It is scikit-learn's default for good reason.
    """
    centroids = [X[rng.integers(len(X))]]
    for _ in range(k - 1):
        d2 = np.min(((X[:, None, :] - np.array(centroids)[None, :, :]) ** 2).sum(2), axis=1)
        centroids.append(X[rng.choice(len(X), p=d2 / d2.sum())])
    return np.array(centroids)

def kmeans(X, k, max_iters=100, seed=0):
    """K-Means clustering (Lloyd's algorithm) with k-means++ initialisation."""
    rng = np.random.default_rng(seed)
    n_samples, _ = X.shape
    centroids = kmeans_plusplus_init(X, k, rng)

    for _ in range(max_iters):
        # Assign each point to its nearest centroid
        distances = np.sqrt(((X - centroids[:, np.newaxis]) ** 2).sum(axis=2))
        labels = np.argmin(distances, axis=0)

        # Update centroids, handling EMPTY clusters.
        # X[labels == i].mean() over an empty selection returns nan, and that nan
        # then propagates into every subsequent distance - silently destroying the
        # run. Reseed an empty cluster to a random point instead.
        new_centroids = []
        for i in range(k):
            points = X[labels == i]
            new_centroids.append(points.mean(axis=0) if len(points)
                                 else X[rng.integers(n_samples)])
        new_centroids = np.array(new_centroids)

        if np.allclose(centroids, new_centroids):
            break
        centroids = new_centroids

    return centroids, labels

def inertia(X, centroids, labels):
    """Within-cluster sum of squares - the objective k-means minimises."""
    return sum(((X[labels == i] - centroids[i]) ** 2).sum() for i in range(len(centroids)))
                

Four caveats that matter more than the code. K-means only reaches a local optimum, so run it several times from different seeds and keep the lowest inertia (scikit-learn's n_init does exactly this). It assumes roughly spherical, similarly-sized clusters, and will confidently carve a crescent or an elongated cluster in half. It requires feature scaling, since it is distance-based. And k must be chosen in advance — use the elbow method on inertia, or silhouette score. If those assumptions do not hold, reach for DBSCAN (density-based, finds arbitrary shapes, infers the cluster count, and labels outliers) or a Gaussian mixture model.

Principal Component Analysis (PCA)

Reduces dimensionality by finding principal components (directions of maximum variance) in the data.

import numpy as np

def pca(X, n_components=2):
    """Principal Component Analysis via eigendecomposition of the covariance matrix."""
    # Center the data (mandatory - PCA finds directions of variance about the mean)
    X_centered = X - np.mean(X, axis=0)

    # Covariance matrix
    cov_matrix = np.cov(X_centered, rowvar=False)

    # eigh, not eig: the covariance matrix is symmetric, and eigh exploits that.
    # It guarantees real eigenvalues and orthonormal eigenvectors, and is faster.
    # eig makes no use of symmetry and can return unordered or complex results.
    eigenvalues, eigenvectors = np.linalg.eigh(cov_matrix)

    # eigh returns ascending order, so reverse for largest-variance-first
    idx = eigenvalues.argsort()[::-1]
    eigenvalues, eigenvectors = eigenvalues[idx], eigenvectors[:, idx]

    components = eigenvectors[:, :n_components]
    X_transformed = X_centered @ components

    # How much of the total variance each component accounts for - always report
    # this, it is how you decide how many components to keep.
    explained_variance_ratio = eigenvalues[:n_components] / eigenvalues.sum()

    return X_transformed, components, explained_variance_ratio


def pca_svd(X, n_components=2):
    """The numerically preferable route: SVD on the centered data directly,
    without ever forming the covariance matrix (which squares the condition
    number). This is what scikit-learn does."""
    X_centered = X - np.mean(X, axis=0)
    U, S, Vt = np.linalg.svd(X_centered, full_matrices=False)
    components = Vt[:n_components].T
    return X_centered @ components, components, (S[:n_components] ** 2) / (S ** 2).sum()
                

Standardise, not just center, when features are on different scales. PCA maximises variance, so a feature measured in millimetres will dominate one measured in metres purely because of its units. Scale to unit variance first unless the units are genuinely comparable.

PCA is linear. For non-linear structure — and especially for visualising clusters in two dimensions — UMAP and t-SNE are the standard tools, though both distort global distances and neither should be used as a preprocessing step for a downstream model.

Neural Networks

Perceptron

The simplest neural network - a single neuron that can perform binary classification.

import numpy as np

class Perceptron:
    def __init__(self, learning_rate=0.01, n_iterations=1000):
        self.learning_rate = learning_rate
        self.n_iterations = n_iterations
        self.weights = None
        self.bias = None
    
    def fit(self, X, y):
        """Train the perceptron"""
        n_samples, n_features = X.shape
        self.weights = np.zeros(n_features)
        self.bias = 0
        
        for _ in range(self.n_iterations):
            for idx, x_i in enumerate(X):
                linear_output = np.dot(x_i, self.weights) + self.bias
                y_predicted = self.activation(linear_output)
                
                # Update weights
                update = self.learning_rate * (y[idx] - y_predicted)
                self.weights += update * x_i
                self.bias += update
    
    def activation(self, x):
        """Step activation function"""
        return np.where(x >= 0, 1, 0)
    
    def predict(self, X):
        """Make predictions"""
        linear_output = np.dot(X, self.weights) + self.bias
        return self.activation(linear_output)
                

Multi-Layer Perceptron (MLP)

A feedforward neural network with multiple layers. Uses backpropagation for training.

Activation functions

Before the code: do not use sigmoid in hidden layers. Its derivative peaks at 0.25 and approaches zero at both tails, so gradients shrink by at least 4× per layer as they propagate backwards. After a handful of layers there is effectively no gradient left and the early layers stop learning. This vanishing gradient problem is the main reason deep networks did not work before around 2012.

Implementation

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-np.clip(z, -250, 250)))

def relu(z):
    return np.maximum(0, z)

class MLP:
    def __init__(self, layers, seed=0):
        rng = np.random.default_rng(seed)
        self.layers = layers
        self.weights, self.biases = [], []
        for i in range(len(layers) - 1):
            # He initialisation: scale by sqrt(2/fan_in) for ReLU layers.
            # Naive small random weights make deep networks converge very slowly;
            # use Xavier/Glorot (sqrt(1/fan_in)) for tanh or sigmoid layers.
            self.weights.append(
                rng.normal(0, np.sqrt(2.0 / layers[i]), (layers[i], layers[i + 1])))
            self.biases.append(np.zeros((1, layers[i + 1])))

    def forward(self, X):
        """Return the pre-activations z as well as the activations a.
           Backprop needs z, so it must be cached here."""
        a = X
        zs, activations = [], [X]
        last = len(self.weights) - 1

        for i in range(len(self.weights)):
            z = a @ self.weights[i] + self.biases[i]
            a = sigmoid(z) if i == last else relu(z)   # ReLU hidden, sigmoid output
            zs.append(z)
            activations.append(a)

        return zs, activations

    def backward(self, zs, activations, y):
        m = y.shape[0]
        n_layers = len(self.weights)
        grad_w = [None] * n_layers
        grad_b = [None] * n_layers

        # Output layer: with a sigmoid output and binary cross-entropy loss, the
        # sigmoid derivative cancels against the loss derivative, leaving just
        # (prediction - target). This cancellation is why the pairing is standard.
        delta = activations[-1] - y

        for i in reversed(range(n_layers)):
            grad_w[i] = activations[i].T @ delta / m
            grad_b[i] = delta.sum(axis=0, keepdims=True) / m

            if i > 0:
                # Propagate back through the ReLU using the PRE-activation z.
                # A common bug is calling a sigmoid-derivative helper on
                # activations[i] - but that value is already sigma(z), so a helper
                # defined as sigmoid(x)*(1-sigmoid(x)) applies sigma a second time
                # and produces silently wrong gradients. Use z, or use a*(1-a).
                delta = (delta @ self.weights[i].T) * (zs[i - 1] > 0)

        return grad_w, grad_b

    def train(self, X, y, epochs=3000, lr=0.5):
        """Full-batch gradient descent. Without this method the class cannot
           learn anything - forward and backward alone never update the weights."""
        for _ in range(epochs):
            zs, activations = self.forward(X)
            grad_w, grad_b = self.backward(zs, activations, y)
            for i in range(len(self.weights)):
                self.weights[i] -= lr * grad_w[i]
                self.biases[i]  -= lr * grad_b[i]

    def predict(self, X):
        return (self.forward(X)[1][-1] > 0.5).astype(int)


# XOR - the classic problem a single perceptron cannot solve
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y = np.array([[0], [1], [1], [0]], dtype=float)

net = MLP([2, 8, 1])
net.train(X, y)
print(net.predict(X).ravel())        # -> [0 1 1 0]
                

Always gradient-check a hand-written backprop. Perturb one weight by a tiny ε, recompute the loss, and compare (L(w+ε) − L(w−ε)) / 2ε against the analytic gradient. They should agree to around 1e−9. Wrong gradients do not usually crash — the network just trains badly, which is far harder to diagnose.

What real training adds

The loop above is plain full-batch gradient descent. Production training adds:

• Adam (or AdamW) instead of vanilla gradient descent — per-parameter adaptive learning rates with momentum. Effectively the default optimizer everywhere.
• Mini-batches rather than the full dataset per step: faster, and the gradient noise itself helps escape poor minima.
• Dropout and weight decay for regularisation, and batch or layer normalisation to stabilise training.
• Learning-rate schedules — warmup then cosine decay is a common recipe.
• Early stopping on a validation split, to halt before the model starts overfitting.

In practice you would write this in PyTorch or JAX rather than NumPy, and get automatic differentiation and GPU execution for free. The value of implementing backprop by hand once is understanding what those frameworks are doing.

Modern Architectures

The MLP above is the foundation, but essentially nothing in production is a plain MLP. The architectures that matter today are specialised to the structure of their data.

Convolutional Neural Networks (CNNs)

For images. Instead of connecting every pixel to every neuron, a CNN slides small learned filters across the image. Two properties follow: parameter sharing (one edge detector works everywhere, so far fewer weights) and translation equivariance (a cat is a cat wherever it appears). Stacked layers build a hierarchy — edges, then textures, then parts, then objects. ResNet's skip connections, which let gradients bypass layers, are what made very deep networks trainable.

Transformers

Introduced in 2017 ("Attention Is All You Need") and now dominant far beyond the translation task they were built for. Transformers underpin GPT, Claude, BERT, and — via vision transformers — much of modern computer vision too.

The core operation is self-attention. Every position produces three vectors: a query (what am I looking for?), a key (what do I offer?), and a value (what do I contribute?). Each position attends to every other by comparing its query against all keys:

Attention(Q, K, V) = softmax(Q Kᵀ / √dₖ) V

  Q Kᵀ        every query dotted with every key -> an n x n relevance matrix
  / √dₖ      scale down, or softmax saturates and gradients vanish
  softmax    normalise each row into weights summing to 1
  ... V      each output is a weighted average of the value vectors

Multi-head attention runs several of these in parallel with different learned projections, letting one head track syntax while another tracks coreference. Because attention is order-agnostic, positional encodings are added to tell the model where each token sits.

Why this displaced RNNs so completely: an RNN processes a sequence one step at a time, so training cannot be parallelised across the sequence, and information from distant tokens must survive many sequential updates. A transformer connects every position to every other in a single operation, and the whole sequence is processed at once — which is what made training on internet-scale corpora feasible. The cost is O(n²) in sequence length, which is why context windows were historically limited and why efficient-attention variants (FlashAttention, sparse and linear attention) are an active area.

RNNs and LSTMs remain worth understanding historically and are still occasionally used for small streaming problems, but they should not be your first choice for sequence modelling. State-space models such as Mamba are a more recent line of work aiming at transformer quality with linear scaling.

Other Architectures

• Graph Neural Networks: for data with explicit relational structure — molecules, social networks, recommendation graphs.
• Diffusion models: generate by learning to reverse a gradual noising process. The basis of modern image and video generation.
• Autoencoders / VAEs: learn compressed representations by reconstructing their own input.

A practical note that cuts across all of this: for tabular data — rows and columns, which is most business data — gradient-boosted trees still generally beat neural networks. Deep learning dominates where the data has strong spatial, sequential, or relational structure that an architecture can exploit.

Evaluation Metrics

Choosing the wrong metric is a more common cause of failed models than choosing the wrong algorithm. Start from the confusion matrix:

                        Predicted
                     Positive  Negative
        Actual  Pos     TP        FN        <- FN: missed a real positive
                Neg     FP        TN        <- FP: false alarm
                

Classification Metrics

• Accuracy: (TP + TN) / total. Misleading under class imbalance — if 1% of transactions are fraudulent, a model that predicts "not fraud" every time scores 99% and is worthless. This is the accuracy paradox, and it is the single most common evaluation mistake.
• Precision: TP / (TP + FP). Of everything flagged, how much was real? Optimise this when false alarms are costly.
• Recall (sensitivity): TP / (TP + FN). Of everything real, how much did we catch? Optimise this when misses are costly — disease screening, fraud detection.
• F1: the harmonic mean of precision and recall. Harmonic, not arithmetic, so it punishes a bad score on either side.
• ROC-AUC: probability that a random positive is ranked above a random negative. Threshold-independent. But note it can look reassuringly high on heavily imbalanced data, because the true-negative pool is enormous.
• PR-AUC (average precision): the one to use for imbalanced problems — it ignores true negatives entirely and reflects performance on the rare class.
• Log loss: penalises confident wrong predictions harshly. The right metric when you care about the probabilities, not just the ranking.

Precision and recall trade off against each other through the decision threshold. The default of 0.5 is arbitrary; choose it deliberately from the cost of each error type.

Regression Metrics

• MAE: mean absolute error. In the units of the target, and robust to outliers.
• MSE / RMSE: squared error penalises large mistakes disproportionately. RMSE is back in the target's units, which makes it easier to interpret than MSE.
• R²: fraction of variance explained. Can go negative if the model is worse than predicting the mean.
• MAPE: mean absolute percentage error. Scale-free and intuitive, but undefined at zero and asymmetric — it punishes over-prediction more than under-prediction.

MAE and RMSE answer different questions. Minimising MAE fits the conditional median; minimising MSE fits the conditional mean. Pick based on whether large errors are disproportionately bad in your application.

Calibration

A model can rank cases perfectly and still output probabilities that are badly wrong. If you take every case where the model said "70%" and only 50% of them turn out positive, the model is miscalibrated — and any decision that uses the probability as a number rather than a rank will be systematically wrong. Accuracy and AUC will not tell you this: both are invariant to any monotonic transform of the scores.

• Reliability diagram: bin predictions by confidence and plot predicted probability against observed frequency. Perfect calibration is the diagonal. Always look at this plot.
• Brier score: mean squared error of the predicted probabilities. Decomposes into calibration and refinement terms.
• Expected Calibration Error (ECE): average gap between confidence and accuracy across bins.

from sklearn.calibration import CalibratedClassifierCV, calibration_curve

# Inspect calibration first
prob_true, prob_pred = calibration_curve(y_valid, model.predict_proba(X_valid)[:, 1],
                                         n_bins=10)
# Plot prob_pred vs prob_true - the diagonal is perfect calibration.

# Then fix it if needed. Fit on data the base model did NOT train on.
calibrated = CalibratedClassifierCV(model, method='isotonic', cv='prefit')
calibrated.fit(X_calib, y_calib)

Two remedies: Platt scaling fits a logistic regression to the model's scores — a two-parameter fix that works well on small calibration sets. Isotonic regression fits an arbitrary monotonic function, which is more flexible but needs more data and can overfit.

Which models need it? Logistic regression is usually well calibrated by construction. Naive Bayes is notoriously overconfident. SVMs produce uncalibrated margins. Boosted trees tend to be over-confident at the extremes. Random forests are conservative — they rarely output values near 0 or 1, because averaging pulls predictions toward the middle. Modern deep networks are generally overconfident.

Validation Strategy

• Three splits, not two. Train fits parameters, validation selects hyperparameters and thresholds, and test is touched exactly once at the end. Tuning against the test set leaks it into the model and the reported score becomes optimistic.
• k-fold cross-validation gives a more stable estimate on small datasets. Use stratified folds for imbalanced classification so each fold keeps the class ratio.
• Respect structure in the data. Time series need forward-chaining splits — a random split lets the model train on the future and predict the past. Grouped data (multiple rows per patient, user, or match) needs group-aware splits, or the same entity appears on both sides.
• Data leakage is the most common cause of a model that scores brilliantly offline and fails in production. Fit scalers, imputers and encoders inside the cross-validation loop, on the training fold only — use a Pipeline so this is automatic. Watch for features that would not exist at prediction time.

Best Practices

• Start with a baseline. Predict the majority class, or the mean. If your sophisticated model cannot beat it, something is wrong. Then try logistic regression or a small gradient-boosted model before anything elaborate.
• Understand the bias-variance trade-off. High bias (underfitting) shows as poor performance on both train and validation; high variance (overfitting) as a large gap between them. The remedies are opposite — more capacity versus more regularisation or data — so diagnose before acting.
• Preprocess inside the pipeline. Fit scalers, imputers and encoders on the training fold only. Fitting on the whole dataset before splitting leaks test information and inflates your scores.
• Three splits. Train, validation, test. Touch the test set once.
• Feature engineering usually beats model selection on tabular problems. Domain knowledge encoded as a feature is worth more than a bigger model.
• Regularization: L2 (ridge) shrinks coefficients smoothly; L1 (lasso) drives them to exactly zero and so performs feature selection. For neural networks, add dropout and early stopping.
• Hyperparameter tuning: random search beats grid search for the same budget, because most hyperparameters do not matter and grid search wastes trials on them. Bayesian optimisation (Optuna) is better still for expensive models.
• Handle class imbalance deliberately: class weights, threshold tuning, or resampling — and evaluate with PR-AUC rather than accuracy.
• Check calibration whenever the probability itself feeds a decision, not just the ranking.
• Set the random seed and record it, or your results are not reproducible.

Real-World Applications

• Image Recognition: Convolutional Neural Networks (CNNs)
• Natural Language Processing: Transformers (RNNs and LSTMs are largely historical here)
• Recommendation Systems: Collaborative filtering, matrix factorization, two-tower neural retrieval
• Tabular Prediction: Gradient-boosted trees — XGBoost, LightGBM, CatBoost
• Fraud Detection: Anomaly detection, classification
• Medical Diagnosis: Classification, pattern recognition
• Autonomous Vehicles: Computer vision, reinforcement learning

What's Next?

Machine learning is a vast field. Continue learning:

• Ensemble Methods - Random forests and gradient boosting, the workhorses of tabular ML
• Modern Architectures - CNNs, transformers and attention
• Trees - Decision trees are built on the tree structures covered there
• Dynamic Programming - The basis of reinforcement learning's value iteration
• Encryption Algorithms - Secure data with cryptography