Ensemble Learning in Machine Learning

Output:
 Ensemble Common
No. Core Idea How It Works Advantages Disadvantages Best Use Cases
Method Algorithms
Train multiple Samples are drawn
models with replacement → - Reduces - Large
Bagging independently on each model gets variance- models may High variance
BaggingClassifier,
1 (Bootstrap random subsets of different data → Handles be slow- May models (Decision
RandomForest
Aggregating) data (with predictions are overfitting- Easy not reduce Trees)
replacement) and averaged or majority to parallelize bias
average/vote results. voted.

Same as bagging but - Reduces

RandomForest - Less
each split in a tree overfitting Classification/reg
Random Bagging + feature Classifier, interpretable
2 considers only a further- Good ression with
Forest randomness. RandomForest than single
random subset of accuracy out-of- tabular data
Regressor tree
features. the-box

Assign weights to
AdaBoost,
Train models samples → - Reduces bias- Highly
Gradient Boosting, - Can overfit-
sequentially, each misclassified samples High accuracy- imbalanced or
3 Boosting XGBoost, Sensitive to
correcting the errors get higher weight → Works well with complex
LightGBM, noise
of the previous one. new model focuses weak learners datasets
CatBoost
on them.

Weak learners
- Simple and
Special case of (shallow trees) trained - Sensitive to
AdaBoost effective- Good Binary/multiclass
4 AdaBoost boosting using sequentially → noisy data &
Classifier for small to classification
exponential loss. weights updated for outliers
medium data
misclassified points.
Fit a model to residual
GradientBoosting, - Handles various Kaggle
Boosting using errors of previous - Slow to train-
Gradient XGBoost, loss functions- competitions,
5 gradient descent on model, using Needs careful
Boosting LightGBM, State-of-the-art high-accuracy
loss function. gradients to minimize tuning
CatBoost accuracy tasks
loss.

Use regularization,
parallelization, - Very fast &
XGBoost / Optimized gradient XGBoost, - Complex
histogram-based accurate- Structured data,
6 LightGBM / boosting LightGBM, tuning- Less
splitting, and Handles large big datasets
CatBoost implementations. CatBoost interpretable
handling of missing datasets well
values.
Combine predictions Train base learners → - Can combine
- Slow When models
of different models use their predictions StackingClassifier, very different
7 Stacking training- Risk capture different
using a meta- as features for a final StackingRegressor algorithms- Often
of overfitting patterns
learner. model. better accuracy
Combine multiple Train models - Simple to - Requires Quick
models by majority independently → vote VotingClassifier, implement- diverse performance
8 Voting
vote or averaging (classification) or VotingRegressor Works with any models for boost with
predictions. average (regression). models benefit minimal effort