Probability¶

Introduction¶

  • Statistics and probability theory constitute a branch of mathematics for dealing with uncertainity. The probability theory provides a basis for the science of statistical inference form data.
  • Sample: (of size n) obtained from a bigger population data to be represented by a probability.
  • Descriptive Statistics: Description of the sample
  • Inferential Statistics: Making decisions / inferences from the sample for our problem


Sample Spaces¶

  • Experiment: Any procedure or process which can have multiple possible outcomes
  • Sample Space: The set of all possible outcomes for a given experiment


Probability Values¶

A set of probability values for an experiment with sample space $S = \{ O_1, O_2, \cdots, O_n \}$ consists of some probabilities that satisfy: \begin{equation} 0 \leq p_i \leq 1, \hspace{0.5cm} i= 1,2, \cdots, n \end{equation} and \begin{equation} p_1 +p_2 + \cdots +p_n = 1 \end{equation} The probability of outcome $O_i$ occurring is said to be $p_i$ and it is written: \begin{equation} P(O_i) = p_i \end{equation} In cases in which the $n$ outcomes are equally likely, then each probability will have a value of $1/n$.


Events and Complements¶

  • Event: Subset of the sample space
  • The probability of an event $A$, $P(A)$, is obtained by the probabilities of the outcomes contained withing the event $A$
  • An event is said to occur if one of the outcomes contained within the event occurs
  • Complement of events: event $ A' $ is the event consisting of everything in the sample space $S$ that is not contained within $A$ \begin{equation} P(A) + P(A ') = 1 \end{equation}
  • Elementary (or simple) event: Event consisting of an individual outcome


Combination of Events¶

  • Intersections of Events: $A \cap B$ consists of the outcomes contained within both events $A$ and $B$
  • Probability of the intersection, $P(A \cap B) $, is the probability that both events occur simultaneously
  • Properties:
    • $P(A \cap B) +P(A \cap B') = P(A)$
    • Mutually exclusive events: if $A \cap B = \emptyset$
    • $A \cap (B \cap C) = (A \cap B) \cap C $


  • Union of Events: $ A \cup B $ consists of the outcomes that are contained within at least one of the events $A$ and $B$
  • The probability of this event, $P (A \cup B)$ is the probability that at least one of these events $A$ and $B$ occurs
  • Properties
    • $P( A \cup B) = P(A \cap B') + P(A' \cap B') + P(A \cap B)$
    • If the events are mutually exclusive, then $P(A \cup B) = P(A) + P(B)$
  • Other simple results:
    • $(A \cup B)' = A' \cap B' $
    • $(A \cap B)' = A' \cup B' $
    • $A \cup A' = S $
    • $A \cup (B \cup C) = (A \cup B) \cup C $


  • $ P(A \cup B \cup C) = P(A) + P(B) + P(C) - P(A \cap B) - P( B \cap C) - P( A \cap C) + P(A \cap B \cap C)$
  • Union of Mutually Exclusive Events: given a sequence $A_1, A_2, \cdots , A_n$ of mutually exclusive events: \begin{equation} P(A_1 \cup A_2 \cup \cdots \cup A_n) = P(A_1) + \cdots + P(A_n) \end{equation}
  • Partition of sample space S: given a sequence $A_1, A_2, \cdots , A_n$ of mutually exclusive events such that $A_1 \cup A_2 \cup \cdots \cup A_n = S$ is called a partition of S


Conditional Probability¶

Definition:

  • Conditional Probability of an event $A$ conditional on an event B is: \begin{equation} P(A \mid B) = \frac{P(A \cap B)}{P(B)} \hspace{0.5cm} \text{for } P(B) >0 \end{equation}

General Multiplication Law:

  • $P (A \mid B) = \frac{P(A \cap B)}{P(B)} \\ \Longrightarrow P(A \cap B) = P(B)P (A \mid B)$
  • $P (A \mid B \cap C) = \frac{P(A \cap B \cap C)}{P(B \cap C)} \\ \Longrightarrow P(A \cap B \cap C) = P(B \cap C)P (A \mid B \cap C)$
  • In general, for a sequence of events $A_1, A_2, \cdots, A_n$: \begin{equation} P(A_1, A_2, \cdots, A_n) = P(A_1)P(A_2 \mid A_1)P(A_3 \mid A_1 \cap A_2) \cdots P(A_n \mid A_1 \cap \cdots \cap A_{n-1}) \end{equation}


Law of Total Probability¶

  • Given $\{ A_1, A_2, \cdots, A_n \}$ a partition of sample space $S$, the probability of an event $B$, $P(B)$ can be expressed as: \begin{equation} P(B) = \sum_{i=1}^n P(A_i)P(B \mid A_i) \end{equation}

Bayes' Theorem¶

  • Given $\{ A_1, A_2, \cdots, A_n \}$ a partition of a sample space, then the posterior probabilities of the event $A_i$ conditional on an event $B$ can be obtained from the probabilities $P(A_i)$ and $P(A_i \mid B)$ using the formula: \begin{equation} P(A_i \mid B) = \frac{P(A_i)P(B \mid A_i)}{\sum_{j=1}^n P(A_j)P(B \mid A_j)} \end{equation}