What is Bayesian classification?
Bayesian classification is a probabilistic approach to predicting the class of a data instance based on Bayes’ theorem. Rather than committing to a single hard rule, it reasons about how likely each class is given the observed features, and assigns the instance to the most probable one. Formally, it computes the posterior probability P(C | X) = [P(X | C) · P(C)] / P(X), where P(C) is the prior (how common the class is before seeing any evidence), P(X | C) is the likelihood (how well the class explains the observed features), and P(X) is the evidence (a normalizing term common to all classes). The classifier then selects the class with the highest posterior — the maximum a posteriori (MAP) decision, which reduces to a maximum likelihood choice when all classes are equally likely.
The most widely used form is the Naïve Bayes classifier, which makes one simplifying assumption: that features are conditionally independent given the class. This lets the likelihood factorize into a simple product of per-feature probabilities, making the method fast and effective even with limited training data. Categorical features are handled through frequency counts, with Laplace smoothing applied to avoid zero probabilities, while continuous features are modelled with a distribution such as the Gaussian density, using each class’s mean and variance (Gaussian Naïve Bayes). Despite the “naïve” independence assumption rarely holding exactly, the method performs remarkably well in practice — powering applications like spam filtering, text and document classification, and medical diagnosis — and serves as the foundation for richer probabilistic models such as Bayesian belief networks.
Points to remember
Bayesian Classifier(Continous Values)
Naïve Bayes is a simple yet powerful probabilistic classifier built on Bayes’ theorem, widely used for its speed, interpretability, and surprisingly strong performance despite its “naïve” assumption that features are conditionally independent given the class. When a feature is continuous — like age or income — the Gaussian variant estimates its likelihood using the normal distribution, fitting a separate mean and standard deviation for each class. This tutorial walks through the complete theory, a fully verified worked numerical example (cross-checked against scikit-learn), and the deeper concepts most often probed in machine learning interviews — from numerical stability tricks to its exact relationship with Logistic Regression and Linear Discriminant Analysis.
The full tutorial — with every calculation shown step by step — is available in the PDF below.
Logistic Regression
Logistic regression is the standard model for predicting a binary outcome — Yes/No, 0/1 — such as whether a customer will buy, a loan will default, or a patient will respond to treatment. Unlike linear regression, it doesn’t predict the outcome directly; it predicts the probability of the outcome by passing a linear score through the sigmoid function, keeping every prediction safely between 0 and 1. Its coefficients are estimated using Maximum Likelihood Estimation, and are interpreted through odds ratios rather than raw units, making the model both accurate and genuinely explainable. This makes logistic regression the default baseline for binary classification and propensity modelling across marketing, healthcare, and finance.
Bayesian Belief Networks
A Bayesian belief network is a compact, graph-based way to represent a joint probability distribution over many random variables. It combines a directed acyclic graph (DAG) — where nodes are variables and edges capture direct probabilistic influence — with a set of local conditional probability distributions (CPDs). Representing a joint distribution explicitly needs an exponential number of parameters (2ⁿ−1 for n binary variables), which is computationally, cognitively, and statistically infeasible beyond a handful of variables. Bayesian networks solve this by exploiting conditional independence: each variable depends directly only on its parents in the graph, letting the full distribution factorize into small local pieces via the chain rule for Bayesian networks — often shrinking the parameter count from exponential to linear.
The graph carries a precise dual meaning: it is simultaneously a scaffold for factorizing a distribution and a compact encoding of independence assumptions, readable directly off the structure through d-separation. This structure supports natural human reasoning patterns — causal reasoning (cause → effect), evidential reasoning (effect → cause), and intercausal reasoning, or “explaining away,” where evidence for one cause lowers belief in a rival cause. The Naïve Bayes classifier is the simplest special case, with one class node and conditionally independent features. Bayesian networks have driven real applications in genetics, medical diagnosis, and decision support — and remain a foundational tool in modern probabilistic and causal reasoning.
The complete tutorial — with worked numerical examples, d-separation, and the theory of I-maps and I-equivalence — is available in the PDF below.
Key Conceptual Challenges in Bayesian Networks
1. Exponential Representation → Compact Factorization
A joint distribution over n binary variables needs 2ⁿ−1 parameters — exponential. A Bayesian network avoids this using a DAG plus conditional-independence assumptions, factorizing:
P(X₁,X₂,X₃,X₄) = P(X₁)P(X₂|X₁)P(X₃|X₁)P(X₄|X₂,X₃)
Key distinction: the chain rule is always valid and gives no compression by itself — compression comes strictly from the conditional-independence structure the graph encodes on top of it. Chain rule = identity; Bayesian-network factorization = chain rule + independence structure.
2. Conditional Independence and d-Separation
(X ⊥ Y | Z) means P(X|Y,Z) = P(X|Z). These independencies are not read off single arrows — they follow from the whole graph via d-separation, through three basic patterns:
- X→Z→Y: conditioning on Z blocks the path.
- X→Z←Y (v-structure): X, Y independent until Z is observed — conditioning creates dependence (explaining away).
- X←Z→Y: Z is a common cause; conditioning on Z blocks the association.
Chain: DAG structure → d-separation → conditional independencies → factorization → inference.
3. Reasoning Types, and the Causality Trap
P(X₁,…,Xₙ) = ∏ᵢ P(Xᵢ|Pa(Xᵢ)) supports causal reasoning (cause→effect), evidential reasoning (effect→cause), and intercausal reasoning / explaining away (one cause’s evidence shifts belief in a rival cause).
Critical distinction: a DAG is fundamentally a probabilistic model — it encodes statistical dependencies, not automatically causal mechanisms. Causal interpretation requires assumptions beyond the graph itself.
4. The Markov Blanket
Parents and children alone do not shield a node from the rest of the network — explaining away means co-parents matter too:
MB(X) = Parents(X) ∪ Children(X) ∪ CoParents(X)
Given MB(X), X is independent of everything else. This underlies Gibbs sampling and local structure learning. A candidate who forgets co-parents hasn’t internalized d-separation.
5. Why Observational Data Can’t Fix Causal Direction
I-equivalent DAGs (same skeleton, same immoralities) encode identical independencies — no statistical test distinguishes X→Y from Y→X within an equivalence class. This is why structure learning outputs a CPDAG, not a single DAG. Recovering true direction needs interventional data or assumptions outside the graph — the exact boundary between a statistical DAG and a causal diagram.
6. NP-Hardness and Treewidth
Exact inference is NP-hard in general; Variable Elimination’s cost is exponential in the induced width of the elimination order — not the variable count directly. A tree-structured network (width 1) is linear-time exact; a dense network can be intractable regardless of ordering, since finding the optimal order is itself NP-hard. The bottleneck is structural (treewidth), not size — this is why approximate inference (loopy BP, sampling) exists.
Neural Networks
Introduction to Neural Networks
This tutorial traces the earliest neural network models through one unifying idea — the straight-line decision boundary. A single neuron computes a weighted sum and fires on one side of the line b + Σwᵢxᵢ = 0, so it can only solve problems that are linearly separable (like AND and OR); XOR is not, and no single unit can learn it. Around this idea the models differ only in how the line is found: the McCulloch–Pitts neuron fixes its weights and threshold by hand with no learning at all; the Hebb net sets them in a single correlational pass (“fire together, wire together”); the perceptron learns by error correction, updating weights only when it misclassifies and converging whenever a separating line exists; and the ADALINE learns by least-squares, using the delta rule (Widrow–Hoff / LMS) to minimise squared error via gradient descent. When one line is not enough, MADALINE stacks two trainable ADALINEs under a fixed OR unit to combine lines and finally solve XOR. Read end to end, it shows the progression from hand-set to trained neurons and sets up the perceptron and the multilayer nets that follow. Download the below file for more detail.
The Perceptron
The perceptron, introduced by Rosenblatt in 1958, is the first neural network that learns its decision line by correcting its own mistakes. Like the earlier models it computes a weighted sum y_in = b + Σwᵢxᵢ and fires on one side of the boundary b + Σwᵢxᵢ = 0, but instead of fixing the weights by hand or in a single pass, it trains iteratively: for each pattern it compares its output with the target, and only when it misclassifies does it nudge the weights and bias by wᵢ += α·t·xᵢ and b += α·t. Correctly classified patterns cause no change, so learning slows as the net improves. Its great guarantee is the perceptron convergence theorem: if the data are linearly separable, the rule is certain to find a separating line in a finite number of steps — though it settles for any correct boundary, not the best-placed one. That same strength is its limit: on non-separable problems such as XOR it never settles, because no single line exists. The perceptron therefore marks the true beginning of trainable neural networks, bridging the hand-set neurons before it and the error-driven, multilayer networks — trained by the delta rule and backpropagation — that follow.
Backpropagation
Backpropagation is the reverse-mode automatic-differentiation algorithm that efficiently computes the gradient of a scalar loss (cost) function with respect to every weight and bias by recursively applying the chain rule across the computational graph. After the forward pass yields each neuron’s pre-activation (net = Σwx + b), activation (out = σ(net)), and the final loss, the backward pass propagates the error signal from output to input, computing at each node a local gradient or delta (∂E/∂net) , the product of the incoming upstream gradient and the activation’s derivative, e.g. o(1−o) for sigmoid — so every weight gradient equals delta × input. These gradients drive gradient-descent optimizers (batch, stochastic, mini-batch SGD, Momentum, RMSProp, Adam), scaled by the learning rate η, updating parameters as w_new = w_old − η ∂E/∂w over successive epochs. Key interview terms include partial derivatives, the Jacobian, credit assignment, and the vanishing/exploding-gradient problem ,caused by repeatedly multiplying small or large derivatives (sigmoid saturation, dead ReLUs) – mitigated by ReLU/tanh activations, Xavier/He initialization, batch normalization, residual connections, and gradient clipping. Overfitting is curbed by regularization (L2/weight decay, dropout), while vectorization and GPU parallelism give speed. For recurrent networks it generalizes to backpropagation-through-time (BPTT). Fundamentally, backpropagation is not a collection of formulas but a single idea : the chain rule applied backward : enabling scalable, end-to-end gradient-based training in modern deep-learning frameworks
Please see the attached solution for complete backpropagation below.
Ensemble learning
Ensemble learning is a machine-learning paradigm that combines many base learners into one stronger predictor, exploiting the bias–variance trade-off: a model’s expected squared error decomposes exactly into bias² (error from oversimplified assumptions, causing underfitting), variance (sensitivity to the training set, causing overfitting), and irreducible noise σ². Ensembles work because averaging M diverse models with pairwise correlation ρ reduces variance as ρσ² + (1−ρ)σ²/M — so both more models and less-correlated (decorrelated) models help. Bagging (Bootstrap Aggregating) trains independent, parallel models on bootstrap samples drawn with replacement (leaving ≈36.8% out-of-bag tuples for free validation, since (1−1/N)^N→1/e) and combines them by majority vote or averaging; it mainly reduces variance, making it ideal for low-bias, high-variance learners like deep decision trees. Random Forests extend bagging by sampling a random feature subset (≈√p) at each split, further decorrelating the trees. Boosting instead builds models sequentially, each re-weighting the tuples the previous one misclassified so later learners focus on hard cases; AdaBoost computes a weighted error, a classifier weight α = ln((1−error)/error), updates tuple weights (with normalization), and predicts by weighted vote, converting weak learners into a strong one by reducing bias. Gradient Boosting (XGBoost, LightGBM) generalizes this by fitting each model to the residual gradient of a differentiable loss. Fundamentally: bagging averages away variance; boosting chips away bias.