Cross-Validation in Machine Learning is an important technique for checking how consistently a predictive model performs on different subsets of data. A model may achieve excellent accuracy on one train-test split but produce weaker results when tested on another group of observations.
Imagine developing a model to predict whether customers will renew a subscription. Evaluating it on only one randomly selected test set may not reveal how sensitive the results are to the chosen records.
Quick Answer: Cross-validation evaluates a machine learning model across multiple data splits. K-Fold divides observations into several folds, while Stratified K-Fold also aims to preserve the proportions of target classes within each fold.
What Is Cross-Validation in Machine Learning?
Cross-validation is a resampling technique used to estimate how well a machine learning model may perform on unseen observations.
Instead of relying on one training-validation split, the available development data is divided into several subsets. The model is trained repeatedly, each time using a different subset for validation.
For example, with five-fold cross-validation, the model is trained and evaluated five times. Each observation serves in a validation fold once.
The resulting scores can be summarised using their mean and standard deviation, helping analysts investigate performance consistency.
Why Is Cross-Validation Important?
Machine learning models can be sensitive to the observations chosen for training and evaluation. One split might contain relatively easy examples, while another contains more challenging records.
Cross-validation helps developers:
- Compare models using several validation splits.
- Investigate variation in model performance.
- Make more informed hyperparameter choices.
- Detect some signs of unstable generalization.
- Use available development data more effectively.
However, cross-validation does not automatically prevent overfitting or eliminate bias. Results still depend on data quality, the splitting strategy and correct preprocessing.
How Does K-Fold Cross-Validation Work?
K-Fold cross-validation divides a dataset into K non-overlapping folds of approximately equal size.
During each iteration, one fold is used for validation and the remaining K-1 folds are used for training.
Consider a dataset containing 100 observations and K = 5. Each fold contains approximately 20 observations.
The model is fitted five times, using four folds for training and one for validation during each iteration.
K-Fold Cross-Validation Example in Python
from sklearn.datasets import load_diabetes
from sklearn.model_selection import KFold, cross_val_score
from sklearn.linear_model import Ridge
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_diabetes(return_X_y=True)
model = make_pipeline(
StandardScaler(),
Ridge(alpha=1.0)
)
kfold = KFold(
n_splits=5,
shuffle=True,
random_state=42
)
scores = cross_val_score(
model,
X,
y,
cv=kfold,
scoring="r2"
)
print("Fold Scores:", scores)
print("Average R2:", scores.mean())The example uses five-fold cross-validation to evaluate a regression model. R-squared measures how well the model explains variation in the target relative to a baseline prediction.
The exact scores depend on the chosen data, model and splitting strategy.
What Is Stratified K-Fold Cross-Validation?
Stratified K-Fold is a variation of K-Fold designed for classification problems.
It attempts to maintain similar proportions of each target class across the folds.
Suppose a customer dataset contains 900 customers who stayed and 100 customers who cancelled.
Ordinary K-Fold does not explicitly preserve this 90:10 target-class distribution. Stratified K-Fold aims to maintain approximately that proportion in each validation fold.
This is particularly useful when some classes appear much less frequently than others.
Stratified K-Fold Python Example
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(
StandardScaler(),
LogisticRegression(max_iter=2000)
)
skf = StratifiedKFold(
n_splits=5,
shuffle=True,
random_state=42
)
scores = cross_val_score(
model,
X,
y,
cv=skf,
scoring="balanced_accuracy"
)
print("Fold Scores:", scores)
print("Mean Score:", scores.mean())
print("Score Variation:", scores.std())This example uses balanced accuracy, which considers recall across the classes. It can be more informative than ordinary accuracy when the class distribution is unequal.
K-Fold vs Stratified K-Fold: Key Differences
| Feature | K-Fold | Stratified K-Fold |
|---|---|---|
| Main purpose | Repeated model evaluation | Class-proportion-aware evaluation |
| Typical application | Regression and other suitable tasks | Classification |
| Class balance | Not explicitly preserved | Approximately preserved |
| Python class | KFold | StratifiedKFold |
| Number of model fits | K | K |
Neither method should be selected automatically. The correct choice depends on the type of prediction problem and the structure of the data.
How Many Folds Should Beginners Use?
Five-fold and ten-fold cross-validation are common starting choices.
Using more folds creates more training and evaluation iterations, increasing computational cost. The best number depends on dataset size, available computing resources and evaluation needs.
For Stratified K-Fold, the number of splits must also be suitable for the number of observations in each target class.
Beginners can start with five folds and compare results when appropriate.
How Do You Avoid Data Leakage During Cross-Validation?
Data leakage occurs when information from validation observations influences the model training process improperly.
For example, scaling an entire dataset before cross-validation allows validation records to influence the scaling statistics.
Scikit-learn Pipelines help avoid this problem by fitting preprocessing operations separately within each training fold.
The Python examples above use make_pipeline() to combine scaling and modelling safely.
For final evaluation, it is also good practice to maintain an independent test set that remains unused during model selection.
When Should You Avoid Ordinary K-Fold?
Ordinary K-Fold may be unsuitable when observations are dependent or must follow a particular order.
For example, a sales forecasting model should generally be evaluated using earlier observations to predict later ones.
TimeSeriesSplit or a suitable chronological evaluation strategy may be more appropriate.
Similarly, if a medical dataset contains repeated measurements from the same patient, group-aware splitting can help ensure that related observations do not appear in both training and validation subsets.
Stratified K-Fold by itself does not solve these time-dependency or group-leakage problems.
Beginner Project: Compare Cross-Validation Results
A simple classification project can demonstrate the practical differences between splitting strategies.
- Load a suitable classification dataset.
- Inspect the distribution of target classes.
- Create five K-Fold splits.
- Create five Stratified K-Fold splits.
- Compare class proportions in the validation folds.
- Evaluate the same model using both methods.
- Compare average scores and score variability.
Document why the scores differ and explain which splitting strategy is more appropriate for the dataset.
Frequently Asked Questions
1. What is Cross-Validation in Machine Learning?
Cross-validation evaluates a model over multiple training-validation splits to estimate performance and investigate how results vary across subsets.
2. What is the difference between K-Fold and Stratified K-Fold?
K-Fold divides observations into folds without explicitly maintaining class proportions. Stratified K-Fold aims to preserve target-class proportions in each fold.
3. Is five-fold cross-validation enough?
Five folds is a practical starting point for many problems, but the appropriate number depends on dataset size, computation and evaluation requirements.
4. Does cross-validation prevent overfitting?
No. It helps evaluate and select models more reliably, but overfitting can still occur. Proper preprocessing and independent final testing remain important.
5. Can Stratified K-Fold be used for regression?
Scikit-learn’s StratifiedKFold is designed for classification labels. Ordinary K-Fold or another suitable strategy is generally used for regression.
Conclusion
Cross-Validation in Machine Learning provides a practical way to evaluate models across multiple data partitions rather than depending on one split.
K-Fold is useful for many evaluation tasks, while Stratified K-Fold is especially relevant for classification when preserving target-class proportions matters.
Beginners should also understand data leakage, pipelines and independent testing before interpreting model scores.
Students who want to develop practical Python, Data Science and Machine Learning skills can explore Full Stack Data Science with Generative AI Training.
Discussion Question: If only 5% of your dataset belongs to the positive class, would you choose K-Fold or Stratified K-Fold for classification?