The central idea is to train a collection of models, often called base classifiers<\/em> (M\u2081, M\u2082, \u2026, M\u2096), and then merge their outputs to form a stronger meta-classifier<\/em> (M*). Instead of relying on a single learning model, the ensemble leverages the collective wisdom of many models.<\/p>\n\n\n\n Given a dataset D<\/strong>, we generate k<\/strong> different training subsets (D\u2081, D\u2082, \u2026, D\u2096). Each subset is used to build one base classifier. When a new data point needs to be classified, each model provides its own prediction (a \u201cvote\u201d). The ensemble aggregates these predictions\u2014often through majority voting\u2014and produces the final decision.<\/p>\n\n\n\n Ensembles typically outperform individual models. For example, if the ensemble relies on majority voting, it will only misclassify a data point if more than half of its base classifiers are wrong. Thus, even when some models make mistakes, the combined decision can still be correct.<\/p>\n\n\n\n The strength of an ensemble lies in diversity<\/strong>\u2014the models should not all make the same types of errors. Low correlation among base classifiers ensures that their mistakes are less likely to overlap. Importantly, each model should perform better than random guessing to contribute meaningfully. Another advantage is that ensemble methods can be parallelized, since each classifier can be trained independently on different processors.<\/p>\n\n\n\n To illustrate, consider a two-class problem with attributes x\u2081<\/em> and x\u2082<\/em>. A single decision tree might generate a rough, piecewise-constant decision boundary. However, when we combine many such trees into an ensemble, the resulting decision surface becomes more refined and closer to the true boundary\u2014demonstrating the increased accuracy of ensembles.<\/p>\n\n\n\n Bagging is one of the simplest yet most effective ensemble techniques. Its goal is to improve prediction accuracy by reducing variance in the models.<\/p>\n\n\n\n Input:<\/strong><\/p>\n\n\n\n Process:<\/strong><\/p>\n\n\n\n This method is called bootstrap aggregation<\/em> because each subset is a bootstrap sample. Since sampling is with replacement, some data points from D may appear multiple times in D\u1d62, while others may be excluded. <\/p>\n\n\n\n Bagging improves robustness, especially against noisy data and overfitting. By averaging across many classifiers, it reduces variance without substantially increasing bias.<\/p>\n\n\n\n A simple analogy is medical diagnosis: instead of asking just one doctor, you consult many doctors. Each gives a possible diagnosis, and the final decision is made based on the majority opinion. This aggregated decision is usually more reliable than relying on a single opinion.<\/p>\n\n\n\n When applied to regression, bagging predicts continuous values by averaging the outputs of the base models rather than voting.<\/p>\n\n\n\n In practice, bagging often delivers a substantial performance boost and forms the basis of more advanced ensemble methods such as random forests<\/strong>.<\/p>\n\n\n\n Boosting is another popular ensemble learning technique that improves accuracy by assigning different weights to classifiers and training samples. Imagine consulting multiple doctors for a diagnosis\u2014not only do you gather their opinions, but you also give more importance (weight) to the doctors who have been more accurate in the past. Similarly, in boosting, classifiers with better performance get stronger influence in the final decision.<\/p>\n\n\n\n This way, boosting builds a set of models that complement one another\u2014later classifiers focus more on the harder cases that earlier ones got wrong.<\/p>\n\n\n\n AdaBoost<\/strong> is the most widely used boosting algorithm. It starts by giving equal weight (1\/d) to all training tuples. Then it proceeds for k<\/em> rounds:<\/p>\n\n\n\n This ensures that difficult-to-classify tuples receive more attention in the next round.<\/p>\n\n\n\n When classifying a new instance X:<\/p>\n\n\n\n As a result, boosting often delivers higher accuracy than bagging, though it can be more sensitive to noisy data.<\/p>\n\n\n\n
\n\n\n\nBagging: Bootstrap Aggregation<\/h1>\n\n\n\n
Algorithm: Bagging<\/h3>\n\n\n\n
\n
\n
a. Generate a bootstrap sample D\u1d62 by sampling from D with replacement<\/em>.
b. Train a classifier M\u1d62 on D\u1d62 using the chosen learning algorithm.<\/li>\n\n\n\n\n
Why Bagging Works?<\/h3>\n\n\n\n
Regression Case<\/h3>\n\n\n\n
Advantages<\/h3>\n\n\n\n
\n
Boosting and AdaBoost<\/strong><\/h5>\n\n\n\n
How Boosting Works<\/h3>\n\n\n\n
\n
\n
\n\n\n\nAdaBoost (Adaptive Boosting)<\/h3>\n\n\n\n
\n
\n
\n\n\n\nWeighted Voting in AdaBoost<\/h3>\n\n\n\n
\n
![\"\"](\"https:\/\/cyberenlightener.com\/wp-content\/uploads\/2025\/09\/image-1.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/cyberenlightener.com\/wp-content\/uploads\/2025\/09\/image-2.png\")
<\/figure>\n\n\n\n\n
\n\n\n\nBagging vs. Boosting<\/h3>\n\n\n\n
\n
Adaboost<\/strong><\/h2>\n\n\n\n