Supervised Learning

Supervised Learning

Supervised learning means teaching a model from examples with known answers. Each training example pairs an input (the features: a house's size, location and age) with the correct output (the label: its sale price). The model looks for the relationship between inputs and outputs, so it can predict the output for inputs it has never seen.

It's called "supervised" because the labels act like a teacher checking every answer during training. Most practical ML in industry is supervised: spam filtering, fraud detection, demand forecasting, credit scoring, image classification.

Two kinds of problems

How a model learns

Every supervised algorithm follows the same loop:

1. Make predictions on training examples with the current parameters.
2. Measure the error with a loss function: how far predictions are from the true labels.
3. Adjust the parameters to reduce the loss.
4. Repeat until the loss stops improving.

For linear regression, the model is a weighted sum, price = w₁·size + w₂·age + b, and the loss is the mean squared error: the average of (prediction − actual)². Gradient descent computes which direction to nudge each weight to reduce the loss, and takes small steps in that direction.

For logistic regression (a classifier, despite the name), the weighted sum is passed through a sigmoid function that squashes it into a probability between 0 and 1. The loss is cross-entropy, which heavily penalizes confident wrong answers.

Common algorithms and when they fit

• Linear / logistic regression: simple, fast and interpretable (each weight shows a feature's influence). A strong baseline. They assume roughly linear relationships.
• Decision trees: learn a flowchart of yes/no questions. Easy to explain, but a single deep tree overfits easily.
• Random forests: many trees trained on random subsets of data and features, with their votes averaged. Robust, with little tuning.
• Gradient-boosted trees (XGBoost, LightGBM): build trees one after another, each correcting the previous ones' mistakes. Usually the top performer on tabular, spreadsheet-like data.
• k-nearest neighbours: predict from the most similar training examples. Simple, but slow at prediction time on large datasets.
• Neural networks: best for unstructured data (images, audio, text), where they learn features themselves. Need more data and compute. See Neural Networks.

Splitting data: train, validation, test

A model's score on the data it trained on says little. It may have memorized it. You need data it hasn't seen:

from sklearn.model_selection import train_test_split

# First hold out a test set that stays locked away until the very end
X_train_full, X_test, y_train_full, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
# Then split the rest into training and validation
X_train, X_val, y_train, y_val = train_test_split(X_train_full, y_train_full, test_size=0.25, stratify=y_train_full, random_state=42)

• Training set: the model fits its parameters here.
• Validation set: you compare models and tune hyperparameters (settings you choose, like tree depth) here.
• Test set: used once, at the end, for an honest estimate. If you keep tweaking until the test score looks good, it's no longer an unbiased estimate.

stratify=y keeps class proportions the same in each split, which is important when one class is rare. For time-based data (sales, prices), split by time (train on the past, test on the future). A random split lets the model "see the future" and flatters it.

Cross-validation makes better use of limited data: split the training data into k folds, train k times, each time validating on a different fold, and average the scores.

Overfitting and underfitting

• Underfitting (high bias): the model is too simple to capture the pattern. Training and validation errors are both high. Fix: a more flexible model, better features, less regularization.
• Overfitting (high variance): the model learns noise and quirks of the training set. Training error is low but validation error is much higher. Fix: more data, simpler model, regularization (penalizing large weights: L1/lasso, L2/ridge), limiting tree depth, early stopping, dropout in neural networks.

This tension is the bias–variance trade-off: more flexibility reduces bias but increases variance. Model selection is choosing the point where validation error is lowest.

Evaluating classifiers properly

Accuracy alone misleads when classes are imbalanced. If 1% of transactions are fraud, a model that always says "not fraud" is 99% accurate and completely useless. Look at the confusion matrix and the metrics built from it:

• Precision = TP / (TP + FP): of the cases flagged positive, how many really were? It matters when false alarms are costly (blocking a legitimate payment).
• Recall = TP / (TP + FN): of the real positives, how many did we catch? It matters when misses are costly (missing a disease or a fraud).
• F1 is the harmonic mean of precision and recall, one number that balances them.
• ROC-AUC measures how well the model ranks positives above negatives across all thresholds. PR-AUC is more informative when positives are rare.

A classifier outputs probabilities, and the threshold (0.5 by default) is a business decision. Lowering it catches more fraud (higher recall) at the cost of more false alarms (lower precision).

For regression: MAE (average absolute error, easy to explain, robust to outliers), RMSE (penalizes large errors more) and R² (the share of variance explained).

Features matter more than algorithms

In practice, quality comes mostly from the data: cleaning errors, handling missing values, encoding categories (one-hot), scaling numeric features for distance- or gradient-based models, and creating informative features ("days since last purchase"). Watch for data leakage, a feature that wouldn't be available at prediction time or that encodes the label ("refund issued" when predicting fraud). It produces spectacular validation scores and a useless model.

Follow-up questions this topic invites — and their answers

Q: Why not just use accuracy? A: On imbalanced data, accuracy rewards predicting the majority class. Use precision/recall/F1 or PR-AUC, and choose based on the cost of each error type: false positives versus false negatives.

Q: How can you tell if a model is overfitting? A: Compare training and validation performance. A large gap (excellent on training, much worse on validation) indicates overfitting. Learning curves, error versus training-set size, also show whether more data would help.

Q: What's the difference between parameters and hyperparameters? A: Parameters are learned from data during training (weights, tree split points). Hyperparameters are chosen by you before training (learning rate, tree depth, regularization strength) and tuned using the validation set or cross-validation.

Q: What is data leakage and how do you prevent it? A: Leakage is when information that wouldn't be available at prediction time gets into training: future data, a feature derived from the label, or preprocessing (like scaling) fitted on the whole dataset before splitting. Prevent it by splitting first, fitting all preprocessing on the training data only (pipelines help), splitting by time for temporal data, and questioning any feature that seems too good.

Q: Why are gradient-boosted trees so popular for tabular data? A: They handle mixed feature types and non-linear interactions, need little feature scaling, are robust to irrelevant features, and consistently achieve top accuracy on structured data with moderate tuning. Neural networks shine on images and text but rarely beat boosted trees on typical business tables.