Machine Learning: 10 Essential Probability Concepts Explained

Probability Concepts for Machine Learning

1. Random Variables

In the field of machine learning, random variables play a crucial role. They represent values whose outcome is uncertain until an event occurs. For example, before receiving an email, one does not know whether it will be classified as spam or not. Similarly, it is unknown whether a visitor on a website will make a purchase until they do. These uncertainties are captured by random variables, which include features, labels, errors, and model outputs.

By convention, random variables are denoted with uppercase letters such as X and Y, while specific observed values are denoted in lowercase, such as x and y. For instance, in a spam classifier, the label can be noted as follows:

• 1 if the email is spam
• 0 if it is not

Thus, Y is a binary variable. Before analyzing the email, it could be either spam or not. Once labeled, the uncertainty disappears, leaving only a number.

In supervised learning, X typically represents the input features and Y the target. The model seeks to answer the following question:

Given what I can observe, what is the probability of each label?

By providing information such as the words present, the sender, or suspicious links, the model can estimate the probability that the email is spam, for example:

P(Y = 1 | X = x) = 0.92

This means there is a 92% chance that this email is spam. It is not a certainty, but a strong probability.

2. Probability Distributions

Once it is known that a variable can take different values, the next question is what values it can take and how often. This distribution is called a probability distribution.

For discrete variables, the probabilities of all possible values must sum to one:

∑ₓ P(X=x) = 1

For continuous variables, it is the area under the curve of the probability density function that must equal one:

∫_{-∞}^{∞} p(x) dx = 1

Different types of data require different distributions. For example, a binary variable like spam/not spam follows a Bernoulli distribution:

Y ~ Bernoulli(p)

where p is the probability that the email is spam.

For continuous values like prediction errors or temperatures, a Gaussian distribution is often used:

X ~ N(μ, σ²)

where μ is the mean and σ² is the variance.

Probability distributions are crucial because machine learning models often attempt to learn them. For example, a regression model tries to estimate probable values for a continuous target, while a classifier estimates a probability distribution over possible classes:

p_θ(y | x)

Here, θ represents the parameters that the model learns during training.

3. Expectation, Variance, and Standard Deviation

When an experiment is repeated multiple times, it is useful to know the expected average value of the results. This average is the expectation or expected value.

For a discrete variable, it is calculated as follows:

E[X] = ∑ₓ x P(X=x)

For a continuous variable:

E[X] = ∫ x p(x) dx

The expectation is useful for evaluating the average performance of a model. For example, if a model predicts house prices, the prediction error will vary from one house to another. The expected error indicates the average error across many predictions.

However, the average alone can be misleading. Two models may have the same average error but very different behaviors. One may have errors close to the average, while the other may have highly dispersed errors. This is where variance comes into play, measuring the spread of values:

Var(X) = E[(X - μ)²], μ = E[X]

The standard deviation, which is the square root of the variance, is often preferred because it is expressed in the same units as the original data:

σ = √Var(X)

4. Conditional Probability

Machine learning models almost never ask questions without context. Instead of simply asking:

What is the probability that an email is spam?

They ask the question in a specific context:

What is the probability that an email is spam given the available information?

This notion of "given" is conditional probability: the probability of an event knowing that another event is already known.

P(A | B) = P(A ∩ B) / P(B)

Classifiers often estimate:

What is the probability of a label given the observed features?

For example, P(Spam | The email contains "free") represents the probability that an email is spam given that it contains the word "free."

If 80% of emails containing "free" are spam, then:

P(Spam | contains "free") = 0.8

The condition modifies the probability. Perhaps only 20% of all emails are spam, but the observation of the word "free" significantly increases that probability. This is how models make predictions: they observe features and update their estimates.

5. Bayes' Theorem

Conditional probability naturally leads to Bayes' theorem, a famous formula in statistics.

Bayes' theorem is often perceived as complex, but it is simply about revising one's beliefs in light of new evidence.

P(A | B) = (P(B | A) P(A)) / P(B)

This formula contains four key elements:

• P(A): the prior belief
• P(B | A): the probability of the evidence if A is true
• P(B): the overall probability of the evidence
• P(A | B): the revised belief after seeing the evidence

Let's take the example of spam. If A is "the email is spam" and B is "it contains the word free," then:

P(Spam | "free") = P("free" | Spam) P(Spam) / P("free")

Even if spam is rare, the presence of the word "free" in an email raises suspicion. This formula adjusts beliefs based on new information. It is used in various contexts, such as Naïve Bayes classifiers or Bayesian neural networks.

6. Joint, Marginal, and Conditional Distributions

So far, we have examined one variable at a time. However, in machine learning, it is often important to understand how multiple variables interact.

When creating a spam detector, for example, one might simultaneously track whether an email contains a link and whether it is spam. The joint distribution is the probability that these two events occur together:

If X is "has a link" and Y is "is spam," then:

P(X=link, Y=spam)

Sometimes, we are only interested in one variable and wish to ignore the other. This is the marginal distribution, obtained by summing the probabilities of the other variables:

P(X) = ∑ᵧ P(X, y)

Finally, the conditional distribution is the probability of one variable given another, already seen above:

P(Y | X) = P(X, Y) / P(X)

These three concepts are interconnected. A model often learns the joint story of the variables and then uses it to estimate the conditional probability.

Independence is a key concept. Two variables are independent when knowing one provides no information about the other:

P(X, Y) = P(X) P(Y)

Although total independence is rare, assuming it exists can simplify a model. Naïve Bayes is a classic example, assuming that features are conditionally independent once the class is known:

P(x₁, x₂, ..., x_d | y) = ∏_{j=1}^{d} P(x_j | y)

Although this assumption is often false, it works surprisingly well, especially for text processing.

7. Likelihood and Maximum Likelihood Estimation

When training a machine learning model, a fundamental question is posed:

How well do the model parameters explain the observed data?

The answer is given by likelihood.

Likelihood measures how probable the observed data is under a given set of model parameters.

Suppose a model with parameters θ assigns probabilities to outcomes:

p_θ(y_i | x_i)

For a dataset containing n independent examples, the likelihood is:

L(θ) = ∏ p_θ(y_i | x_i)

This amounts to multiplying the probabilities assigned by the model to all training examples. A model that consistently assigns a high probability to the correct outcomes will have a high likelihood.

This leads to a training strategy called Maximum Likelihood Estimation (MLE).

The idea is to choose the parameter values that make the observed data as probable as possible:

θ̂_MLE = argmax L(θ)

In practice, multiplying many probabilities can yield very small numbers, which are difficult for computers to handle.