Lecture-by-lecture study notes for Statistics 110, beginning with probability and counting.
Statistics 110: Probability, taught by Joe Blitzstein at Harvard University, introduces probability as a language for understanding uncertainty, randomness, and statistical reasoning. The course develops both intuition and mathematical problem-solving skills, beginning with counting and sample spaces before moving to conditional probability, Bayes’ rule, random variables, probability distributions, expectation, limit theorems, and Markov chains.
Follow the course video lectures on YouTube, or visit the official course website for supporting materials and practice problems.
Probability begins with a precise description of possible outcomes. For finite, equally likely outcomes, probability reduces to counting. The multiplication rule and combinations provide the tools to count poker hands and distinguish the four basic sampling cases.
| Symbol | Meaning |
|---|---|
| \(S\) | Sample space: all possible outcomes |
| \(A\subseteq S\) | Event: a subset of the sample space |
| \(P(A)\) | Probability of event \(A\) |
| \(\lvert S\rvert\), \(\lvert A\rvert\) | Number of outcomes in the sample space and the event |
| \(n!\) | Factorial; \(0!=1\) |
| \(\binom nk\) | Number of unordered selections of \(k\) distinct objects from \(n\) |
| \(n\) and \(k\) | Number of available objects or types, and sample size |
| \(A\cup B\), \(A\cap B\), \(A^c\) | Union, intersection, and complement |
After studying these notes, you should be able to:
Probability provides a mathematical way to describe uncertainty. Applications of probability and statistics include physics, genetics, economics, game theory, finance, history, government, and everyday reasoning.
Frederick Mosteller and David Wallace investigated whether Alexander Hamilton or James Madison wrote 12 disputed Federalist essays. They compared word frequencies in texts with known authorship, focusing on common words whose use is relatively stable across topics. These writing habits provided evidence for comparing the two possible authors using statistical methods, including Bayesian inference. Their analysis supported Madison as the author of all 12 disputed essays.
Core idea: Uncertainty can concern a fixed historical fact. The author does not change, but our assessment of who it was can change as we examine evidence. Probability provides a way to quantify that uncertainty and update it.
Games involving dice, coins, and cards have clearly defined outcomes, making them useful for developing probability models. In their 1654 correspondence, Pierre de Fermat and Blaise Pascal explored games of chance, including how to divide a prize fairly when a contest stops before either player wins. Their letters helped establish the foundations of modern probability.
Supplementary illustration: For a simplified example, suppose Alice and Bob have equal chances of winning each independent round. The first to win three rounds receives a $100 prize, but play stops with Alice leading two wins to one. Alice needs one more win; Bob needs two.
Imagine two further rounds, including an unused round if Alice wins immediately. The four equally likely sequences of round winners are:
| Next two round winners | Winner of the contest |
|---|---|
| Alice, Alice | Alice |
| Alice, Bob | Alice |
| Bob, Alice | Alice |
| Bob, Bob | Bob |
Alice wins the contest in three of the four sequences, so her chance of winning is \(3/4\); Bob’s is \(1/4\). Dividing the prize in proportion to these chances gives Alice \(75\) and Bob \(25\).
Core idea: A fair division reflects each player’s chance of winning from the current position. Counting equally likely future sequences turns that idea into a precise calculation. The imagined unused round keeps all sequences the same length without changing the contest’s winner.
Main lesson: Intuition can be unreliable in probability. Precise definitions and explicit reasoning help us check it.
An experiment is any process with possible outcomes whose result is not known in advance. It need not be a laboratory experiment.
Examples include tossing a coin twice, rolling two dice, or selecting five cards.
An outcome is one possible result. The sample space, written as \(S\), is the set of all possible outcomes.
For two coin tosses:
\[S=\{HH,HT,TH,TT\}.\]The positions represent the first and second tosses, so \(HT\) and \(TH\) are different outcomes.
For two distinguishable six-sided dice:
\[S=\{(i,j):i,j\in\{1,2,3,4,5,6\}\}, \qquad \lvert S\rvert=6\times6=36.\]Here, \(\lvert S\rvert\) means the number of elements in \(S\). You can distinguish the dice by color, or distinguish the first roll from the second.
An event is a subset of the sample space:
\[A\subseteq S.\]An event occurs when the observed outcome belongs to that subset.
For two coin tosses:
| Event | Subset of the sample space |
|---|---|
| Both tosses are tails | \(\{TT\}\) |
| Exactly one head | \(\{HT,TH\}\) |
| At least one head | \(\{HH,HT,TH\}\) |
Unions, intersections, and complements translate statements about events into precise set notation:
| Notation | Meaning |
|---|---|
| \(A\cup B\) | \(A\) or \(B\) occurs, including the possibility that both occur |
| \(A\cap B\) | Both \(A\) and \(B\) occur |
| \(A^c\) | \(A\) does not occur; the outcomes in \(S\) outside \(A\) |
| \(\varnothing\) | The empty event |
| \(A\subseteq B\) | Every outcome in \(A\) also belongs to \(B\) |
For a finite sample space whose outcomes are equally likely:
\[\boxed{P(A)=\frac{\lvert A\rvert}{\lvert S\rvert} =\frac{\text{number of favorable outcomes}}{\text{number of possible outcomes}}.}\]“Favorable” means that the outcome satisfies event \(A\), whether or not that outcome is desirable.
Without these conditions, simply counting favorable and possible outcomes does not establish the probability.
If \(HH,HT,TH,TT\) are equally likely, then:
\[P(\text{two tails})=\frac{1}{4}.\]Only one of the four outcomes belongs to the event.
Equal chances of heads and tails on an individual toss do not, by themselves, guarantee that the four two-toss sequences are equally likely. Dependence between tosses can change their probabilities.
Supplementary illustration: Suppose the first toss is fair, but the second result always repeats the first. Each position individually has a 50% chance of heads, yet:
\[P(HH)=P(TT)=\frac12,\qquad P(HT)=P(TH)=0.\]Independent fair tosses would make all four sequences equally likely. Independence is a concept developed more fully later in probability.
Consider the claim: “There either is life on Neptune or there is not, so the probability of life is 1/2.” This argument is invalid because listing two possibilities does not establish that they are equally likely.
Key distinction: Not knowing the probability is different from knowing that outcomes are equally likely.
Intelligent life is a more restrictive condition than life of any kind. If \(I\) is the event of intelligent life and \(L\) is the event of any life, then:
\[I\subseteq L\quad\Longrightarrow\quad P(I)\le P(L).\]Strict inequality requires positive probability for life without intelligent life; it does not follow from subset inclusion alone.
For two fair, independent dice, the 36 ordered pairs are equally likely. The possible sums \(2,3,\ldots,12\) are not equally likely.
There is one pair giving sum 2, but six pairs giving sum 7. Therefore:
\[P(\text{sum}=2)=\frac1{36},\qquad P(\text{sum}=7)=\frac6{36}.\]Suppose a process has \(r\) stages. There are \(n_1\) choices at stage 1, \(n_2\) choices at stage 2 for each possible first choice, and so on. If stage \(i\) always has \(n_i\) choices after any allowed preceding history, then:
\[\boxed{\text{Number of complete outcomes}=n_1n_2\cdots n_r.}\]There are two cone types and three flavors available with either cone:
Choose a cone
├── Cake
│ ├── Chocolate
│ ├── Vanilla
│ └── Strawberry
└── Waffle
├── Chocolate
├── Vanilla
└── Strawberry
Each complete path represents a different cone–flavor pair. There are:
\[2\times3=6\]possible pairs. Choosing the flavor before the cone gives the same count: \(3\times2=6\).
Ten stages with two choices each produce:
\[2^{10}=1024\]outcomes. Listing every outcome quickly becomes impractical.
For a positive integer \(n\):
\[n!=n(n-1)(n-2)\cdots2\cdot1\]For example, \(4!=24\). The convention is \(0!=1\).
Select \(k\) distinct objects from \(n\), keeping track of selection order.
The multiplication rule gives:
\[n(n-1)\cdots(n-k+1)=\frac{n!}{(n-k)!}.\]If order does not matter, the ordered count counts each group repeatedly. Every group of \(k\) distinct objects has exactly \(k!\) orderings.
Divide by that common overcounting factor:
\[\boxed{\binom nk=\frac{n!}{k!(n-k)!}.}\]This is a binomial coefficient, read as “\(n\) choose \(k\).” It counts subsets of size \(k\) from \(n\) distinct objects.
For \(0\le k\le n\), use the formula above. For nonnegative integers with \(k>n\), the count is zero because the selection is impossible.
Choose two people from \(A,B,C\).
Ordered selections:
AB, BA, AC, CA, BC, CB
Unordered groups:
{A,B}, {A,C}, {B,C}
Thus:
\[\binom32=\frac{3\times2}{2!}=3.\]Important: Division by \(k!\) works here because every selected object is distinct and every group was counted exactly \(k!\) times.
A standard deck has 52 distinct cards: 13 ranks, each with four suits. A five-card hand is an unordered selection without replacement.
Assume every five-card hand is equally likely.
A full house contains exactly three cards of one rank and two cards of a different rank, such as three 7s and two 10s.
| Choice | Number of possibilities | Explanation |
|---|---|---|
| Rank of the three-card group | \(13\) | Any rank is possible |
| Three suits for that rank | \(\binom43=4\) | Choose three of its four cards |
| Rank of the two-card group | \(12\) | It must differ from the first rank |
| Two suits for that rank | \(\binom42=6\) | Choose two of its four cards |
By the multiplication rule:
\[\lvert A\rvert=13\binom43\cdot12\binom42 =13\cdot4\cdot12\cdot6 =3744.\]Each full house has a unique triple rank, pair rank, and selection of suits. The procedure therefore counts every full house exactly once.
That is approximately 0.1441%, or about one in 694 uniformly random five-card hands.
Two questions determine the basic counting method:
For \(n\) distinct available objects or types and a sample of size \(k\):
| Sampling | Order matters | Order does not matter |
|---|---|---|
| With replacement | \(n^k\) | \(\binom{n+k-1}{k}\) |
| Without replacement | \(\frac{n!}{(n-k)!}\) | \(\binom nk\) |
Assume \(n\ge1\) and \(k\ge0\). Without replacement, the displayed factorial expressions apply when \(k\le n\); the count is zero if \(k>n\).
Each of the \(k\) positions has \(n\) choices:
\[n\times\cdots\times n=n^k.\]Supplementary example: A four-digit code, allowing leading zeros and repeated digits, has \(10^4=10,000\) possibilities.
Each selection removes an object:
\[n(n-1)\cdots(n-k+1)\]Supplementary example: Awarding gold, silver, and bronze to three of eight people, with no ties, gives \(8\times7\times6=336\) possibilities.
Select a group, ignoring order:
\[\binom nk.\]Supplementary example: A three-person committee from eight people can be chosen in \(\binom83=56\) ways.
Repetitions are allowed, but order is ignored. Such a selection is called a multiset.
The number of such selections is:
\[\binom{n+k-1}{k}.\]A small case helps illustrate what the formula counts:
Supplementary small-case check: Choose two items from \(A,B,C\), allowing repetition and ignoring order:
AA, AB, AC, BB, BC, CC
There are six possibilities, agreeing with \(\binom{3+2-1}{2}=\binom42=6\).
Why you cannot simply divide \(n^k\) by \(k!\) — supplementary
With repetition, different unordered selections can have different numbers of ordered versions. For example, \(AB\) corresponds to \(AB\) and \(BA\), but \(AA\) has only one ordering.
There is no uniform \(k!\) overcounting factor.
The sampling table counts possibilities; it does not automatically make them equally likely.
For two independent uniform draws from \(A,B,C\), there are nine equally likely ordered sequences. After ignoring order:
\[P(AA)=\frac19,\qquad P(\{A,B\})=\frac29.\]Although there are six multisets, they are not equally likely under this sampling process. Consequently, dividing by six would give incorrect probabilities for this experiment.
Try these before reading the answers.
The four sequences are equally likely, so \(P(A)=2/4=1/2\).
Each drink can be paired with any of three pastries, giving \(4\times3=12\) pairs.
There are four choices for each of five positions, giving \(4^5=1024\) strings.
There are seven choices for president and six remaining choices for secretary, giving \(7\times6=42\) assignments. The roles matter.
Order does not matter, so the count is:
\[\binom73=\frac{7\cdot6\cdot5}{3\cdot2\cdot1}=35.\]All unordered pairs of distinct balls are equally likely under the stated sampling process. There are \(\binom82\) total pairs and \(\binom52\) red pairs:
\[P(\text{both red})=\frac{\binom52}{\binom82} =\frac{10}{28}=\frac5{14}.\]Use the formula for unordered selections with replacement:
\[\binom{4+3-1}{3}=\binom63=20.\]This is a count, not a claim that those selections are equally likely under every drawing process.
The possibilities must also be equally likely before favorable-outcome counting gives \(1/2\). The number of labels alone supplies no evidence of equal likelihood.
Any of 13 ranks can supply the triple. The pair must use one of the other 12 ranks. There is no division by 2 because the triple and pair roles distinguish the two ranks; each full house is already counted once.
| Concept | Essential fact |
|---|---|
| Sample space | Set of possible outcomes, \(S\) |
| Event | Subset \(A\subseteq S\) |
| Naive probability | \(P(A)=\lvert A\rvert/\lvert S\rvert\) for finite, equally likely outcomes |
| Multiplication rule | Multiply stage counts when each stage has the stated number of choices after every allowed history |
| Ordered, with replacement | \(n^k\) |
| Ordered, without replacement | \(n!/(n-k)!\) |
| Unordered, without replacement | \(\binom nk\) |
| Unordered, with replacement | \(\binom{n+k-1}{k}\) |
| Full house | \(13\binom43\cdot12\binom42/\binom{52}{5}\approx0.1441\%\) |
The set of all possible outcomes of the experiment.
A subset of the sample space; it occurs when the observed outcome belongs to that subset.
Outcomes assigned the same probability by the model. This assumption must be justified before applying the counting ratio.
A counting principle that multiplies the numbers of choices at successive stages, provided each stage has the stated count after every allowed preceding history.
An unordered selection of distinct objects, counted by a binomial coefficient.
Returning a selected object to the available pool so that it can be selected again.
An unordered selection that permits repeated objects or types.
A five-card hand with three cards of one rank and two cards of a different rank.
Counting becomes easier when a problem is represented in a useful way. Labeling clarifies which outcomes are distinct; stars and bars turns repeated selections into arrangements of symbols; story proofs establish identities by counting the same collection in two ways. Probability axioms then extend the framework to unequal probabilities and infinite sample spaces.
| Symbol | Meaning |
|---|---|
| \(n\) | Number of available types or labeled boxes in a stars-and-bars problem |
| \(k\) | Number of selections or identical objects to distribute |
| \(x_i\) | Number of selections of type \(i\), or occupancy of box \(i\) |
| \(\binom{n+k-1}{k}\) | Number of unordered selections with replacement |
| \(m,n\) | Sizes of two disjoint groups in Vandermonde’s identity |
| \(j\) | Number selected from the first group |
| \(S\) | Sample space |
| \(P(A)\) | Probability assigned to event \(A\) |
| \(\varnothing\) | Empty event |
| \(A_i\cap A_j=\varnothing\) | Events \(A_i\) and \(A_j\) are disjoint |
| \(\bigcup_{j=1}^{\infty}A_j\) | Event that at least one of the events \(A_1,A_2,\ldots\) occurs |
After studying these notes, you should be able to:
Suppose a jar contains three red balls and one green ball, and each physical ball is equally likely to be drawn. Label them \(R_1,R_2,R_3,G_1\). The sample space for one draw has four equally likely outcomes, so:
\[P(\text{red})=\frac34.\]The color labels “red” and “green” describe only two possible observations, but those observations are not equally likely. Red combines three underlying outcomes, whereas green corresponds to one.
Core idea: Labels make the elementary outcomes explicit. Ignoring a distinction in the final observation does not erase its effect on probability.
Choose the four-person team. Everyone else belongs to the six-person team:
\[\binom{10}{4}=210.\]Choosing the six-person team first gives the same partition, so:
\[\binom{10}{4}=\binom{10}{6}.\]There is no division by two: each partition has exactly one four-person team, so choosing that team counts each partition once.
Now split ten people into two five-person teams with no labels or different roles assigned to the teams.
Choosing five people gives \(\binom{10}{5}\) possibilities, but it counts each partition twice. Selecting one team first or selecting its complement first produces the same pair of teams.
Therefore:
\[\boxed{\text{Number of partitions}=\frac12\binom{10}{5}=126.}\]If the teams instead have distinct labels, such as Team A and Team B, assigning a group to Team A differs from assigning that group to Team B. The count is then \(\binom{10}{5}=252\).
When can you divide? Divide by a number \(c\) only after showing that every desired outcome was counted exactly \(c\) times. A correction that varies from outcome to outcome cannot be applied as one common divisor.
Choose \(k\) times from \(n\) available types, allowing repetition and ignoring selection order. Assume \(n\ge1\) and \(k\ge0\).
Instead of recording a sequence, record how often each type was chosen:
\[(x_1,x_2,\ldots,x_n),\qquad x_i\ge0,\qquad x_1+\cdots+x_n=k.\]The entries are nonnegative integers. For example, choosing types \(A,A,C\) gives counts \((2,0,1)\) for types \(A,B,C\). Sequences \(AAC\), \(ACA\), and \(CAA\) all give the same count vector.
The following descriptions refer to the same collection of possibilities:
| Description | What one outcome records |
|---|---|
| Unordered sampling with replacement | How many times each type was selected |
| Identical objects in labeled boxes | How many objects occupy each box |
| Nonnegative integer solutions | A vector \((x_1,\ldots,x_n)\) whose entries sum to \(k\) |
Each type corresponds to a box, and each selection adds one identical tally mark to that box. The boxes remain distinct because they represent different types; the tally marks do not need identities because order is ignored.
| Case | Direct reasoning | Formula check |
|---|---|---|
| \(k=0\) | One empty selection | \(\binom{n-1}{0}=1\) |
| \(k=1\) | Choose any one of the \(n\) types | \(\binom n1=n\) |
| \(n=1\) | Every selection is of the only type | \(\binom kk=1\) |
| \(n=2\) | The first count can be \(0,1,\ldots,k\); the second is determined | \(\binom{k+1}{k}=k+1\) |
These checks help interpret the formula. They do not replace a proof for arbitrary \(n\) and \(k\).
The number of ways to distribute \(k\) identical objects among \(n\) labeled boxes, allowing empty boxes, is:
\[\boxed{\binom{n+k-1}{k}=\binom{n+k-1}{n-1}.}\]Equivalently, this counts nonnegative integer solutions of \(x_1+\cdots+x_n=k\) and unordered selections of size \(k\) with replacement from \(n\) types.
Use a star * for each object and a bar | between neighboring boxes.
For four boxes containing \((3,0,2,1)\) objects, the encoding is:
Box 1 Box 2 Box 3 Box 4
*** | | ** | *
Compact encoding: ***||**|*
There are six stars and three bars. The adjacent bars indicate an empty second box. A leading bar would indicate an empty first box, and a trailing bar would indicate an empty last box.
The four boxes contain \((1,2,3,1)\) particles, so \(n=4\) and \(k=7\). Reading from left to right, each dot becomes a star and each boundary between boxes becomes a bar:
*|**|***|*
The lower row also shows an outer wall at each end. Those two walls are fixed; they are not extra separators to arrange. Removing them leaves seven stars and three internal bars, or ten positions. Thus all configurations of seven identical objects in four labeled boxes are counted by:
\[\binom{10}{7}=\binom{10}{3}=120.\]Core idea: The box boundaries preserve the order of the labeled boxes, while the dots record only occupancies. The diagram uses seven particles; the earlier \((3,0,2,1)\) example uses six and shows how an empty box is encoded.
Every occupancy vector produces exactly one string. Conversely, count the stars before the first bar, between successive bars, and after the last bar to recover every box count.
This is a bijection: a one-to-one correspondence between the two collections. Nothing is omitted and nothing is counted twice.
A valid string contains:
Choose which \(k\) positions contain stars. All remaining positions contain bars:
\[\text{Number of strings}=\binom{n+k-1}{k}.\]Choosing the \(n-1\) bar positions instead gives \(\binom{n+k-1}{n-1}\). Both descriptions specify the same strings.
For the example: Six objects among four boxes give \(\binom96=\binom93=84\) possible occupancy vectors.
Why not choose gaps only between the stars? That would rule out adjacent bars and bars at the ends, excluding empty boxes. Stars and bars allows those arrangements because a type may be selected zero times.
Why not divide \(n^k\) by \(k!\)? Different count vectors have different numbers of ordered versions. For example, \(AAA\) has one ordering, while \(AAB\) has three. There is no common \(k!\) overcounting factor.
Distribute seven identical objects into three labeled boxes, with at least one in each box.
First place one object in each box. Four objects remain, and they may be distributed with empty boxes allowed. Thus:
\[\binom{3+4-1}{4}=\binom64=15.\]More generally, for \(k\ge n\ge1\), positive integer solutions of \(x_1+\cdots+x_n=k\) are counted by:
\[\binom{k-1}{n-1}.\]If \(k<n\), no such solution exists. The change of variables \(y_i=x_i-1\) explains the result: the \(y_i\) are nonnegative and sum to \(k-n\).
The stars-and-bars count is also called the Bose–Einstein value, reflecting its connection to counting occupation configurations of indistinguishable particles. For ordinary sampling problems, however, counting configurations does not establish that those configurations are equally likely.
The four equally likely ordered outcomes are:
\[HH,\quad HT,\quad TH,\quad TT.\]Ignoring order leaves three head–tail count configurations:
| Configuration | Ordered outcomes represented | Probability |
|---|---|---|
| Two heads | \(HH\) | \(1/4\) |
| One head and one tail | \(HT,TH\) | \(1/2\) |
| Two tails | \(TT\) | \(1/4\) |
Stars and bars correctly counts the three configurations:
\[\binom{2+2-1}{2}=3.\]It does not imply that each has probability \(1/3\). A model assigning equal probabilities to the three configurations would describe a different random experiment.
Core idea: Whether we can visually distinguish objects is separate from how the experiment assigns probabilities. Identical-looking coins still have a first and second toss, or can be labeled coin 1 and coin 2.
The physics connection has limits. Bose–Einstein counting concerns indistinguishable particles and their occupation configurations. It does not make ordinary coin-toss configurations uniformly distributed, nor does the counting formula alone determine a physical system’s probabilities.
For \(k\) independent draws, each uniformly choosing one of \(n\) types, every ordered sequence has probability \(1/n^k\).
A count vector \((x_1,\ldots,x_n)\) represents:
\[\frac{k!}{x_1!x_2!\cdots x_n!}\]ordered sequences. To see why, arrange the \(k\) selections and divide out the permutations within each repeated type. Therefore:
\[P(\text{counts }(x_1,\ldots,x_n)) =\frac{k!}{x_1!\cdots x_n!}\frac1{n^k}.\]This explains precisely why different occupancy vectors generally receive different probabilities.
A story proof establishes a mathematical result through an interpretation. For a counting identity, interpret both sides as the number of objects in the same collection, then justify each count.
A story proof is a general argument. Verifying a few numerical examples provides checks, but does not prove an identity for all admissible values.
For integers \(0\le k\le n\):
\[\boxed{\binom nk=\binom n{n-k}.}\]Choose a committee of \(k\) people from \(n\) people. Specifying the members uniquely determines the \(n-k\) nonmembers, and specifying the nonmembers uniquely determines the members.
The two sides count the same committees through complementary descriptions.
Core idea: Choosing what to include is equivalent to choosing what to exclude.
For integers \(1\le k\le n\):
\[\boxed{n\binom{n-1}{k-1}=k\binom nk.}\]Count committees of size \(k\) with one committee member designated as president.
| Method | First choice | Second choice | Total |
|---|---|---|---|
| President first | One of \(n\) people | Remaining \(k-1\) members from the other \(n-1\) | \(n\binom{n-1}{k-1}\) |
| Committee first | A \(k\)-person committee from \(n\) | President from its \(k\) members | \(\binom nk\,k\) |
Every committee–president pair is counted exactly once by each method, proving the identity.
For example, selecting a three-person committee with a president from five people gives:
\[5\binom42=30=3\binom53.\]Core idea: Changing the order in which we specify a complete outcome can produce a useful identity without changing the outcomes being counted.
For nonnegative integers \(m,n,k\) with \(k\le m+n\):
\[\boxed{\binom{m+n}{k} =\sum_{j=0}^{k}\binom mj\binom n{k-j}.}\]A binomial coefficient is zero when its lower argument exceeds its nonnegative upper argument. Impossible selections therefore contribute zero to the sum.
There are two disjoint groups: the first has \(m\) people, and the second has \(n\) people. Choose a committee of size \(k\).
Count directly: Choose any \(k\) of the total \(m+n\) people:
\[\binom{m+n}{k}.\]Count by composition: Suppose exactly \(j\) committee members come from the first group. Then \(k-j\) must come from the second group. For this fixed \(j\), the multiplication rule gives:
\[\binom mj\binom n{k-j}.\]Add over all possible \(j\). The cases are disjoint because a committee has exactly one value of \(j\), and they cover all committees. This proves the identity.
The feasible values are:
\[\max(0,k-n)\le j\le\min(k,m).\]Using \(j=0,\ldots,k\) is convenient because the infeasible terms are already zero.
Choose three people from a group of three and a separate group of four:
| Number from the first group, \(j\) | Number from the second group | Number of committees |
|---|---|---|
| 0 | 3 | \(\binom30\binom43=4\) |
| 1 | 2 | \(\binom31\binom42=18\) |
| 2 | 1 | \(\binom32\binom41=12\) |
| 3 | 0 | \(\binom33\binom40=1\) |
Hence:
\[4+18+12+1=35=\binom73.\]Core idea: Multiply choices within a fixed case; add counts across disjoint cases.
If every \(k\)-person committee is equally likely, then:
\[P(\text{exactly }j\text{ from the first group}) =\frac{\binom mj\binom n{k-j}}{\binom{m+n}{k}}.\]Vandermonde’s identity guarantees that these probabilities sum to 1. It connects a combinatorial identity to the requirement that all possible cases account for the whole experiment.
The formula \(P(A)=\lvert A\rvert/\lvert S\rvert\) requires a finite sample space with equally likely outcomes. To handle unequal probabilities or infinitely many outcomes, assign probabilities to events through a function \(P\).
At this introductory level, a probability space consists of:
The input to \(P\) is an event, which is a set of outcomes. Its output is a number.
The outer rectangle is \(S\). Event \(A\) contains five of the nine outcomes; event \(B\) contains four. One outcome belongs to both, so \(A\) and \(B\) are not disjoint.
If all nine outcomes are equally likely, each has probability \(1/9\). Then:
\[P(A)=\frac59,\qquad P(B)=\frac49,\qquad P(A\cap B)=\frac19,\qquad P(A\cup B)=\frac89.\]Adding \(5/9\) and \(4/9\) counts the shared pebble twice. Their union contains eight distinct pebbles, so direct addition is invalid here.
For a general finite model, imagine giving the pebbles nonnegative masses totaling 1. The probability of an event is the sum of the masses inside it. Equal masses recover counting; unequal masses require adding the assigned probabilities instead. The picture specifies which outcomes belong to each event, but the size or number of drawn circles alone does not specify their probabilities.
Core idea: Combining disjoint events adds separate masses. When events overlap, their common outcomes must be accounted for only once.
The empty event contains no outcome, so it cannot occur. The full sample space contains every possible outcome, so it must occur.
If \(A_1,A_2,\ldots\) are pairwise disjoint events, then:
\[\boxed{P\left(\bigcup_{j=1}^{\infty}A_j\right) =\sum_{j=1}^{\infty}P(A_j).}\]“Pairwise disjoint” means:
\[A_i\cap A_j=\varnothing\qquad\text{whenever }i\ne j.\]No outcome belongs to two of the events. Their union therefore combines non-overlapping probability contributions.
Countable additivity also gives finite additivity: set all events after the last one to \(\varnothing\). In particular, if \(A\cap B=\varnothing\), then:
\[P(A\cup B)=P(A)+P(B).\]Disjointness is essential. For two independent fair coin tosses, let \(A\) mean the first toss is heads and \(B\) mean the second toss is heads. The outcome \(HH\) belongs to both events. Adding \(P(A)+P(B)=1\) counts it twice, whereas \(P(A\cup B)=3/4\).
Independent events are not necessarily disjoint. Independence describes how probabilities relate; disjointness means that the events cannot occur together.
Let \(S=\{a,b,c\}\), with individual probabilities \(0.2,0.3,0.5\). For any event, add the probabilities of its outcomes. Then:
\[P(\{a,c\})=0.2+0.5=0.7.\]The empty set has total probability 0, the full space has total probability 1, and disjoint sets add without double counting. This is a valid probability model, although counting two favorable outcomes out of three would give the wrong answer.
Equal masses of \(1/N\) on a finite sample space of size \(N\) recover the naive formula. Thus the counting definition is a special case of the general framework.
Let \(S=\{1,2,3,\ldots\}\) and assign:
\[P(\{j\})=2^{-j},\qquad j=1,2,\ldots.\]The total probability is the geometric series:
\[\sum_{j=1}^{\infty}2^{-j}=1.\]For example, the probability of an even result is:
\[P(\{2,4,6,\ldots\})=\sum_{r=1}^{\infty}4^{-r}=\frac13.\]The outcomes are not equally likely, but the model has a well-defined total probability. No division by an infinite number of outcomes is needed.
Technical clarification for infinite spaces: In a fully general treatment, probabilities are assigned to a specified collection of measurable events, and a probability space is written \((S,\mathcal F,P)\). For finite or countable spaces, we can take all subsets as events. For uncountable spaces, such as a continuous interval, additional care is needed; not every subset must be assigned a probability.
Probability zero is not always impossibility. The empty event always has probability zero, but in continuous models a particular point can also have probability zero. The reverse implication is not generally valid.
Try these before reading the answers.
For teams of three and five, choose the three-person team:
\[\binom83=56.\]For two unnamed teams of four, each partition is counted twice by selecting a four-person team:
\[\frac12\binom84=35.\]For labeled teams A and B, choosing Team A uniquely determines Team B, giving \(\binom84=70\) assignments.
The string is:
**||*|**
There are five stars and three bars, so:
\[\binom{4+5-1}{5}=\binom85=56.\]For nonnegative entries:
\[\binom{3+6-1}{6}=\binom82=28.\]For positive entries, subtract one from each variable. The new nonnegative entries sum to 3, giving:
\[\binom{3+3-1}{3}=\binom52=10.\]There are \(\binom{2+3-1}{3}=4\) configurations. The eight ordered sequences are equally likely:
| Counts \((x_A,x_B)\) | Number of ordered sequences | Probability |
|---|---|---|
| \((3,0)\) | 1 | \(1/8\) |
| \((2,1)\) | 3 | \(3/8\) |
| \((1,2)\) | 3 | \(3/8\) |
| \((0,3)\) | 1 | \(1/8\) |
The configurations are not uniform because they represent different numbers of equally likely ordered outcomes.
Choose the president from six people, then two other members from the remaining five: \(6\binom52\).
Alternatively, choose three members from six, then select one of the three as president: \(3\binom63\).
Both count the same committee–president pairs, and both equal 60.
Count by the number selected from the first group:
\[\binom40\binom53+\binom41\binom52+ \binom42\binom51+\binom43\binom50 =10+40+30+4=84=\binom93.\]Exactly two from the first group gives \(\binom42\binom51=30\) committees, so:
\[P(\text{exactly two})=\frac{30}{84}=\frac5{14}.\]Their union is \(S\), so \(P(A\cup B)=1\). Adding 0.5 and 0.9 counts outcome \(b\) twice. Additivity in its direct form requires disjoint events.
By countable additivity:
\[P(\{3,4,5,\ldots\})=\sum_{j=3}^{\infty}2^{-j} =\frac{1/8}{1-1/2}=\frac14.\]| Concept | Essential fact |
|---|---|
| Labels | Identify elementary outcomes before grouping them into observations |
| Overcounting correction | Divide by \(c\) only when every desired outcome is counted exactly \(c\) times |
| Stars and bars | \(k\) identical objects, \(n\) labeled boxes, empty boxes allowed: \(\binom{n+k-1}{k}\) |
| Positive occupancies | For \(k\ge n\ge1\): \(\binom{k-1}{n-1}\) |
| Complement identity | \(\binom nk=\binom n{n-k}\) |
| Committee–president identity | \(n\binom{n-1}{k-1}=k\binom nk\) |
| Vandermonde’s identity | \(\binom{m+n}{k}=\sum_{j=0}^k\binom mj\binom n{k-j}\) |
| Story proof | Justify two counts of the same collection |
| Probability function | Maps events to numbers in \([0,1]\) |
| Normalization | \(P(\varnothing)=0\) and \(P(S)=1\) |
| Countable additivity | For pairwise disjoint events, probability of the union equals the sum of probabilities |
| Main modeling warning | Equally likely sequences can produce unequally likely count configurations |
A vector \((x_1,\ldots,x_n)\) recording the number of objects in each labeled box. For unordered repeated selections, it records how many times each type was chosen.
A one-to-one correspondence between two collections. Every element of either collection has exactly one partner in the other, so finite collections related by a bijection have the same size.
An encoding of identical objects as stars and boundaries between labeled boxes as bars. Choosing the star or bar positions gives the number of nonnegative occupancy vectors.
A proof by interpretation. In combinatorics, it often establishes an identity by counting the same collection in two justified ways.
An equality between choosing a committee from two combined groups and summing over all possible ways to split its membership between those groups.
A sample space together with a probability function on its events. In the fully general formulation, the collection of measurable events is also specified.
Events for which every pair of distinct events has empty intersection. At most one of them can occur in a single outcome.
The probability of a union of countably many pairwise disjoint events equals the sum of their individual probabilities.
A coincidence can become likely when there are many opportunities for it to occur. The birthday problem illustrates this through counting the complement. Probability axioms give general rules for complements, containment, and overlapping events. Inclusion–exclusion then handles the overlaps in a shuffled deck, giving a surprisingly stable probability of at least one matching card.
| Symbol | Meaning |
|---|---|
| \(k\) | Number of people in the birthday problem |
| \(M\), \(M^c\) | At least one birthday match, and no birthday matches |
| \(\prod_{j=0}^{k-1}(1-j/365)\) | Product of the \(k\) factors for distinct birthdays |
| \(A^c\) | Complement of event \(A\) within \(S\) |
| \(B\cap A^c\) | Outcomes in \(B\) but outside \(A\) |
| \(A\subseteq B\) | Every outcome in \(A\) belongs to \(B\) |
| \(\bigcup_{i=1}^n A_i\) | At least one of the events \(A_1,\ldots,A_n\) occurs |
| \(\bigcap_{i\in I}A_i\) | All events whose indices belong to \(I\) occur |
| \(n\) | Number of distinct cards in the matching problem |
| \(\pi(i)\) | Label on the card in position \(i\) of a shuffled deck |
| \(A_i=\{\pi(i)=i\}\) | Card \(i\) matches its position |
| \(D_n\) | Number of permutations of \(n\) objects with no fixed points |
| \(e\) | Base of the natural logarithm; \(e\approx2.71828\) |
After studying these notes, you should be able to:
Suppose \(k\) people are in a room. What is the probability that at least two have the same birthday?
Use the following idealized model:
An outcome is an ordered list of \(k\) birthdays. Each position has 365 possibilities, so:
\[\lvert S\rvert=365^k.\]The uniformity and independence assumptions make every such list equally likely, with probability \(365^{-k}\).
Let \(M\) be the event of at least one match. A match may involve one pair, several pairs, or three or more people on one day. All of these belong to \(M\).
Modeling clarification: Real birthdays need not be uniformly distributed or independent. The formula below is exact for the stated model; it is not an exact description of every real group. Labeling people does not establish independence—it specifies which outcomes we are counting.
With zero or one person, a match is impossible. With more than 365 people, a match is certain by the pigeonhole principle: assigning more than 365 people to 365 days forces some day to receive at least two people.
With exactly 365 people, a match is extremely likely but not certain: an assignment with one person on each day is still possible.
Directly counting all possible kinds of matches is difficult because the cases overlap. The complement \(M^c\) has a simple description: all birthdays are different.
For \(1\le k\le365\):
| Person | Allowed birthdays if all birthdays must differ |
|---|---|
| First | 365 |
| Second | 364 |
| Third | 363 |
| \(k\)th | \(365-k+1\) |
Thus:
\[\lvert M^c\rvert=365\cdot364\cdots(365-k+1).\]Divide by the number of equally likely birthday lists:
\[P(M^c)=\frac{365\cdot364\cdots(365-k+1)}{365^k} =\prod_{j=0}^{k-1}\left(1-\frac j{365}\right).\]For \(k=0\), the empty product is 1, giving probability 0. For \(k>365\), use \(P(M)=1\) rather than extending the product to negative factors.
Core idea: The experiment allows repeated birthdays, so the denominator counts sampling with replacement. The no-match event restricts birthdays to distinct days, so its numerator counts assignments without replacement. Both counts still describe ordered lists for the same labeled people.
For 23 people:
\[P(M)=1-\frac{365\cdot364\cdots343}{365^{23}} \approx0.507297.\]The no-match probability is approximately \(0.492703\). For 22 people, the match probability is approximately \(0.475695\), so 23 is the smallest group size with a match probability above 50%.
| People, \(k\) | Probability of at least one match |
|---|---|
| 2 | \(0.002740\) |
| 10 | \(0.116948\) |
| 22 | \(0.475695\) |
| 23 | \(0.507297\) |
| 30 | \(0.706316\) |
| 50 | \(0.970374\) |
| 57 | \(0.990122\) |
| 100 | \(0.999999693\) |
A group of 23 people contains:
\[\binom{23}{2}=253\]pairs. The event concerns any of these pairs, not just the 22 comparisons involving one particular person. The number of pairs grows quadratically with group size.
Incorrect: There are 253 pairs, each matching with probability \(1/365\), so the probability of a match is \(253/365\).
Correction: Pair-match events overlap. If three people share a birthday, all three pairs among them match. Adding their probabilities counts that outcome repeatedly. The ratio \(253/365\approx0.693151\) is not the probability of at least one match.
Fix your birthday and consider 22 other people under the same independent, uniform model. Each avoids your birthday with probability \(364/365\). Therefore:
\[P(\text{at least one shares your birthday}) =1-\left(\frac{364}{365}\right)^{22} \approx0.058571.\]This is about 5.86%, compared with 50.73% for any match among all 23 people. The first event asks for a match with one specified birthday; the second allows any pair to match.
A probability function assigns every event a number in \([0,1]\). The axioms are:
\[P(\varnothing)=0,\qquad P(S)=1,\]and, for pairwise disjoint events:
\[P\left(\bigcup_{i=1}^{\infty}A_i\right) =\sum_{i=1}^{\infty}P(A_i).\]These rules apply beyond finite, equally likely sample spaces. The following properties are consequences of the axioms, rather than additional assumptions.
The events \(A\) and \(A^c\) are disjoint, and their union is \(S\). By additivity:
\[1=P(S)=P(A\cup A^c)=P(A)+P(A^c).\]Rearranging gives:
\[\boxed{P(A^c)=1-P(A).}\]This justifies the final step in the birthday calculation. It also applies when individual outcomes have unequal probabilities or when the sample space is infinite.
Core idea: An event and its complement divide the entire experiment into two mutually exclusive possibilities, whose probabilities sum to 1.
If \(A\subseteq B\), then:
\[\boxed{P(A)\le P(B).}\]Separate \(B\) into the outcomes in \(A\) and the outcomes outside \(A\):
\[B=A\cup(B\cap A^c).\]These two pieces are disjoint. Hence:
\[P(B)=P(A)+P(B\cap A^c)\ge P(A),\]because probabilities are nonnegative.
Roll a fair six-sided die. Let \(A=\{6\}\) and \(B=\{4,5,6\}\). Since \(A\subseteq B\):
\[P(A)=\frac16\le\frac36=P(B).\]The extra outcomes \(B\cap A^c=\{4,5\}\) contribute probability \(2/6\).
Containment need not give strict inequality. Even if \(A\) is a proper subset of \(B\), it is possible that \(P(A)=P(B)\). Strict inequality holds exactly when \(P(B\cap A^c)>0\). A nonempty event need not have positive probability in every model.
For any two events \(A\) and \(B\):
\[\boxed{P(A\cup B)=P(A)+P(B)-P(A\cap B).}\]The sum \(P(A)+P(B)\) counts outcomes in the intersection twice. Subtracting \(P(A\cap B)\) leaves each outcome in the union counted once.
Write the union as two disjoint pieces:
\[A\cup B=A\cup(B\cap A^c).\]Therefore:
\[P(A\cup B)=P(A)+P(B\cap A^c).\]Also, \(B=(A\cap B)\cup(B\cap A^c)\) is a disjoint union, so:
\[P(B\cap A^c)=P(B)-P(A\cap B).\]Substitution gives the addition rule. No independence assumption is needed. For disjoint events, the intersection has probability zero and the rule reduces to direct addition.
For a fair die, let \(A=\{2,4,6\}\) and \(B=\{4,5,6\}\). Their intersection is \(\{4,6\}\). Hence:
\[P(A\cup B)=\frac36+\frac36-\frac26=\frac46=\frac23.\]The union is \(\{2,4,5,6\}\), confirming the result by direct counting.
Core idea: “Or” includes outcomes where both events occur, but each such outcome contributes only once to the union.
Track how often a particular outcome contributes:
| Events containing the outcome | Single-event additions | Pairwise subtractions | Triple addition | Net count |
|---|---|---|---|---|
| Exactly one | 1 | 0 | 0 | 1 |
| Exactly two | 2 | 1 | 0 | 1 |
| All three | 3 | 3 | 1 | 1 |
The pairwise intersections include the triple intersection. Subtracting all three pairwise terms removes a central outcome three times after adding it three times, leaving zero. The final addition restores its contribution to one.
For events \(A_1,\ldots,A_n\):
\[\boxed{ P\left(\bigcup_{i=1}^{n}A_i\right) =\sum_{r=1}^{n}(-1)^{r+1} \sum_{1\le i_1<\cdots<i_r\le n} P(A_{i_1}\cap\cdots\cap A_{i_r}). }\]Add single-event probabilities, subtract two-event intersections, add three-event intersections, and continue with alternating signs. Each unordered group of indices appears once; the condition \(i_1<\cdots<i_r\) prevents repeated listings of the same intersection.
Core idea: Inclusion–exclusion corrects overlap systematically. Symmetry makes it especially useful when all intersections involving the same number of specified events have the same probability.
Incorrect: For three events, subtracting the three pairwise intersections is enough.
Correction: The triple intersection must be added back. Pairwise intersections mean that both named events occur, whether or not a third event also occurs; they do not mean “exactly those two.”
Shuffle \(n\ge1\) distinct cards labeled \(1,2,\ldots,n\), with all \(n!\) permutations equally likely. Reveal them in order while counting \(1,2,\ldots,n\). Win if at least one revealed card matches the number called out.
A match is a fixed point of the permutation. If the deck is \((3,2,1,4)\), positions 2 and 4 match; positions 1 and 3 do not.
Let:
\[A_i=\{\pi(i)=i\}.\]The event of winning is \(A_1\cup\cdots\cup A_n\). The events overlap because several positions can match at once.
Fix card \(i\) in position \(i\). The remaining \(n-1\) cards may be arranged freely, giving \((n-1)!\) favorable permutations:
\[P(A_i)=\frac{(n-1)!}{n!}=\frac1n.\]For distinct positions \(i_1,\ldots,i_r\), fixing their cards leaves \((n-r)!\) arrangements:
\[P(A_{i_1}\cap\cdots\cap A_{i_r})=\frac{(n-r)!}{n!}.\]For example:
\[P(A_i\cap A_j)=\frac1{n(n-1)},\qquad P(A_i\cap A_j\cap A_\ell)=\frac1{n(n-1)(n-2)}.\]Fixing the specified positions does not forbid additional matches among the remaining positions. That is exactly what an intersection requires: all named events occur, possibly along with others.
There are \(\binom nr\) ways to choose the \(r\) specified matching positions. Each intersection has the same probability, so the total at order \(r\) is:
\[\binom nr\frac{(n-r)!}{n!} =\frac{n!}{r!(n-r)!}\frac{(n-r)!}{n!} =\frac1{r!}.\]Therefore:
Core idea: A complicated union becomes manageable because intersection probabilities depend only on how many positions are specified. The binomial coefficient counts the choices of positions; the factorial ratio counts decks satisfying those choices.
For \(n=3\):
\[P(\text{win})=1-\frac12+\frac16=\frac23.\]All six decks can be listed:
| Deck order | Matching positions | Result |
|---|---|---|
| \((1,2,3)\) | 1, 2, 3 | Win |
| \((1,3,2)\) | 1 | Win |
| \((2,1,3)\) | 3 | Win |
| \((2,3,1)\) | None | Lose |
| \((3,1,2)\) | None | Lose |
| \((3,2,1)\) | 2 | Win |
Four of six permutations win. No permutation has exactly two matching positions: if two cards are fixed, the remaining card is forced into its own position.
Incorrect: Each position matches with probability \(1/n\), so the probability of winning is \(n(1/n)=1\).
Correction: The events are not disjoint. The fully ordered deck belongs to every \(A_i\) and is counted \(n\) times in that sum. Inclusion–exclusion corrects those repeated contributions.
Another trap: The match events are not independent. For \(n\ge2\), \(P(A_i\cap A_j)=1/[n(n-1)]\), whereas \(P(A_i)P(A_j)=1/n^2\).
A derangement is a permutation with no fixed points. It is a losing deck in the matching game.
Taking the complement of the winning probability gives:
\[P(\text{no matches})=\sum_{r=0}^{n}\frac{(-1)^r}{r!}.\]Multiply by the total number of permutations:
\[\boxed{D_n=n!\sum_{r=0}^{n}\frac{(-1)^r}{r!}.}\]For example, \(D_3=6(1-1+1/2-1/6)=2\), agreeing with the two losing decks above.
The exponential series gives:
\[e^{-1}=\sum_{r=0}^{\infty}\frac{(-1)^r}{r!}.\]Consequently:
\[\boxed{P(\text{no matches})\longrightarrow e^{-1}\approx0.367879,}\] \[\boxed{P(\text{at least one match})\longrightarrow1-e^{-1}\approx0.632121.}\]| Cards, \(n\) | Exact winning probability | Decimal |
|---|---|---|
| 1 | \(1\) | \(1.000000\) |
| 2 | \(1-1/2!\) | \(0.500000\) |
| 3 | \(1-1/2!+1/3!\) | \(0.666667\) |
| 4 | \(1-1/2!+1/3!-1/4!\) | \(0.625000\) |
| 5 | \(1-1/2!+1/3!-1/4!+1/5!\) | \(0.633333\) |
| 6 | \(1-1/2!+1/3!-1/4!+1/5!-1/6!\) | \(0.631944\) |
The finite probabilities alternate around the limit; they do not increase monotonically with deck size. A larger deck provides more candidate matching positions, but each specified position matches with smaller probability \(1/n\).
The alternating-series remainder bounds the error by the first omitted term:
\[\left|P(\text{win})-(1-e^{-1})\right|\le\frac1{(n+1)!}.\]For six cards, the error is at most \(1/7!\approx0.000198413\). For 52 distinct numbered cards, the limiting value is an extraordinarily accurate approximation, although the finite alternating sum remains the exact probability.
Try these before reading the answers.
The denominator counts all birthday assignments with repetition allowed. The no-match numerator counts only assignments of distinct birthdays:
\[P(M)=1-\frac{365\cdot364\cdot363}{365^3} =\frac{1093}{133225}\approx0.008204.\]The probability is about 0.8204%. Taking the complement includes every possible kind of match without counting overlapping match cases separately.
The smallest size is 366. With 365 people, assigning exactly one person to each day produces no match. With 366 people, there are more people than available days, so at least two must share a day.
All ten people avoid your birthday with probability \((364/365)^{10}\), so:
\[P(\text{at least one shares yours}) =1-\left(\frac{364}{365}\right)^{10} \approx0.027062.\]This is about 2.7062%. Any match among eleven people also includes pairs among the other ten, so it is a different, larger event.
The disjoint decomposition of \(B\) gives:
\[P(B\cap A^c)=P(B)-P(A)=0.60-0.25=0.35.\]The complement rule gives \(P(B^c)=1-0.60=0.40\).
Neither event is the complement of the union, so its probability is \(0.10\).
Exactly one consists of the disjoint events \(A\cap B^c\) and \(B\cap A^c\):
\[P(\text{exactly one})=(0.60-0.20)+(0.50-0.20)=0.70.\]The union includes the both-events case; exactly one excludes it.
Omitting the triple correction gives \(0.75\), undercounting the union by the probability \(0.05\) of the triple intersection.
Fix cards 1 and 3 in positions 1 and 3. Arrange the remaining three cards in \(3!=6\) ways. Thus:
\[P(A_1\cap A_3)=\frac{3!}{5!}=\frac1{20}.\]Other positions may also match. The intersection requires at least the two specified matches, not exactly two matches.
The no-match probability is \(3/8\), so:
\[D_4=4!\cdot\frac38=9.\]Each specified position matches with probability \(1/4\). But fixing two specified cards leaves \(2!\) favorable permutations out of \(4!\):
\[P(A_1\cap A_2)=\frac{2!}{4!}=\frac1{12}.\]Multiplying \(1/4\) by \(1/4\) would require independence, which does not hold for these events.
| Concept | Essential fact |
|---|---|
| Birthday sample space | \(365^k\) equally likely ordered assignments under uniformity and mutual independence |
| No birthday match | \(\prod_{j=0}^{k-1}(1-j/365)\) for \(1\le k\le365\) |
| At least one birthday match | One minus the no-match probability; exceeds 50% first at \(k=23\) |
| Guaranteed birthday match | \(k>365\) |
| Complement | \(P(A^c)=1-P(A)\) |
| Monotonicity | \(A\subseteq B\Rightarrow P(A)\le P(B)\) |
| Two-event addition | \(P(A\cup B)=P(A)+P(B)-P(A\cap B)\) |
| Inclusion–exclusion | Add singles, subtract pairs, add triples, and continue alternating |
| \(r\) specified card matches | \((n-r)!/n!\) |
| Total order-\(r\) contribution | \(\binom nr(n-r)!/n!=1/r!\) |
| Matching-game win | \(\sum_{r=1}^n(-1)^{r+1}/r!\) |
| Derangements | \(D_n=n!\sum_{r=0}^n(-1)^r/r!\) |
| Large-deck win probability | \(1-e^{-1}\approx0.632121\) |
An event in which at least two people share a birthday. It includes several pairs or larger groups sharing a day, not only exactly one matching pair.
Placing more objects than boxes into boxes forces at least one box to contain two or more objects.
The probability that an event does not occur is one minus the probability that it occurs.
If one event is contained in another, its probability cannot exceed that of the containing event.
A formula for the probability of a union that corrects repeated contributions from overlapping events using alternating sums of intersection probabilities.
An ordering of distinct objects. There are \(n!\) permutations of \(n\) distinct objects.
A position \(i\) at which a permutation leaves the label unchanged: \(\pi(i)=i\). In the matching game, it is a card whose label equals its position.
A permutation with no fixed points. In the matching game, it corresponds to a deck with no matching positions.
Independence specifies when intersection probabilities can be multiplied. The Newton–Pepys problem combines independence, counting, and complements to compare three dice games. Conditional probability updates a probability when evidence is known: restrict attention to outcomes consistent with that evidence, then renormalize. Bayes’ rule relates the two directions of conditioning through the same joint probability.
| Symbol | Meaning |
|---|---|
| \(A\cap B\) | Both events \(A\) and \(B\) occur |
| \(P(A\mid B)\) | Probability of \(A\) given that \(B\) occurred; requires \(P(B)>0\) |
| \(P(A\cap B)=P(A)P(B)\) | Definition of independence of two events |
| \(A^c\) | Event that \(A\) does not occur |
| \(n\), \(r\) | Number of dice and number of sixes in a dice calculation |
| \(\binom nr\) | Number of choices of the \(r\) dice that show six |
| \(H\) | A hypothesis whose probability is being updated |
| \(E\) | Observed evidence |
| \(P(H)\) | Prior probability of the hypothesis |
| \(P(E\mid H)\) | Likelihood: probability of the evidence if the hypothesis holds |
| \(P(H\mid E)\) | Posterior probability after conditioning on the evidence |
After studying these notes, you should be able to:
Two events \(A\) and \(B\) are independent if:
\[\boxed{P(A\cap B)=P(A)P(B).}\]The definition is symmetric: interchanging \(A\) and \(B\) changes neither side. It also makes sense when either event has probability zero.
Independence means that information about one event does not change the probability of the other, whenever the relevant conditional probability is defined. It is a property of events under a probability model, not something established merely by giving the events different names.
Let \(A\) be the event that the first toss is heads and \(B\) the event that the second toss is heads. The four sequences \(HH,HT,TH,TT\) are equally likely. Thus:
\[P(A)=P(B)=\frac12,\qquad P(A\cap B)=P(\{HH\})=\frac14.\]Since \(1/4=(1/2)(1/2)\), the events are independent. They can occur together: \(HH\) belongs to both.
Disjointness is the set statement \(A\cap B=\varnothing\): the events cannot occur together. Independence is the probability statement \(P(A\cap B)=P(A)P(B)\).
If disjoint events both have positive probability, then:
\[P(A\cap B)=0<P(A)P(B),\]so they are dependent. Learning that one occurred rules out the other.
For example, on one fair die, “roll a 1” and “roll a 2” are disjoint, but \(0\ne(1/6)(1/6)\). By contrast, the two heads events above are independent and overlap.
Incorrect: Independent events have nothing in common, so they cannot occur together.
Correction: Independent events may occur together. Their joint probability equals the product of their individual probabilities. Disjoint events can be independent only if at least one has probability zero.
If \(A\) and \(B\) are independent, then \(A\) and \(B^c\) are independent. To prove this, split \(A\) into two disjoint pieces:
\[A=(A\cap B)\cup(A\cap B^c).\]Therefore:
\[\begin{aligned} P(A\cap B^c) &=P(A)-P(A\cap B)\\ &=P(A)-P(A)P(B)\\ &=P(A)(1-P(B))\\ &=P(A)P(B^c). \end{aligned}\]Interchanging the roles of the events also proves that \(A^c\) and \(B\) are independent. Applying the same result again shows that \(A^c\) and \(B^c\) are independent.
Core idea: Knowing whether \(B\) occurs gives the same information as knowing whether \(B^c\) occurs. If that information does not affect \(A\), changing the description to a complement does not create dependence.
For three events \(A,B,C\), mutual independence requires all four conditions:
\[\begin{aligned} P(A\cap B)&=P(A)P(B),\\ P(A\cap C)&=P(A)P(C),\\ P(B\cap C)&=P(B)P(C),\\ P(A\cap B\cap C)&=P(A)P(B)P(C). \end{aligned}\]If only the first three hold, the events are pairwise independent. Checking each pair is not enough to establish mutual independence.
For \(n\) events, mutual independence requires the product rule for every subset of two or more events:
\[P\left(\bigcap_{i\in I}A_i\right)=\prod_{i\in I}P(A_i), \qquad I\subseteq\{1,\ldots,n\},\quad\lvert I\rvert\ge2.\]Toss two fair, independent coins. Let:
Each event has probability \(1/2\), and each pairwise intersection is \(\{HH\}\), with probability \(1/4\). Every pair therefore satisfies the independence equation.
But:
\[P(A\cap B\cap C)=\frac14\ne\frac18=P(A)P(B)P(C).\]Knowing either toss alone does not determine whether the tosses agree. Knowing both tosses determines agreement completely.
Core idea: Information from several events together may be useful even when information from each event separately is not.
Compare three games, using fair six-sided dice with mutually independent results:
| Game | Number of dice | Winning event |
|---|---|---|
| \(A\) | 6 | At least one six |
| \(B\) | 12 | At least two sixes |
| \(C\) | 18 | At least three sixes |
Which game is most likely to win?
More dice give more opportunities for sixes, but the winning threshold also increases. These events belong to different experiments; no containment argument orders their probabilities.
An outcome is an ordered list of \(n\) die results. There are \(6^n\) equally likely lists.
To obtain exactly \(r\) sixes:
The number of favorable outcomes is \(\binom nr5^{n-r}\), so:
\[\boxed{P(\text{exactly }r\text{ sixes}) =\frac{\binom nr5^{n-r}}{6^n} =\binom nr\left(\frac16\right)^r\left(\frac56\right)^{n-r}.}\]This calculation uses counting and independence directly. The same expression will later appear as a Binomial probability.
The complement has no sixes. Each die then has five allowed results:
\[P(A)=1-\frac{5^6}{6^6} =1-\left(\frac56\right)^6 \approx0.665102.\]The complement has either zero sixes or exactly one six. These cases are disjoint:
\[\begin{aligned} P(B) &=1-\frac{5^{12}+\binom{12}{1}5^{11}}{6^{12}}\\ &=1-\left(\frac56\right)^{12} -12\left(\frac16\right)\left(\frac56\right)^{11}\\ &\approx0.618667. \end{aligned}\]The factor 12 chooses which die shows the single six. The other eleven dice must all avoid six.
The complement has zero, one, or two sixes:
\[\begin{aligned} P(C) &=1-\frac{5^{18}+\binom{18}{1}5^{17}+\binom{18}{2}5^{16}}{6^{18}}\\ &\approx0.597346. \end{aligned}\]For exactly two sixes, \(\binom{18}{2}\) chooses their positions. Each of the other sixteen dice has five possible non-six results.
The six-dice game has the largest winning probability, about 66.51%, compared with 61.87% and 59.73%.
Core idea: Count the few losing cases rather than all the winning cases. Independence determines the probability of each ordered outcome; combinations account for where the sixes appear.
Incorrect: Group twelve dice into two groups of six. At least two sixes means each group must contain a six, so the winning probability is \(P(A)^2\).
Correction: Both sixes may lie in the same group. Requiring one in each group describes a smaller event and misses valid wins. Splitting eighteen dice into three groups produces the same problem.
In 1693 the diarist Samuel Pepys asked Isaac Newton this question. Pepys strongly believed that game \(C\) was the most likely, but could not support it with mathematics. Newton’s calculation showed that game \(A\) is the most likely. Newton computed the probabilities correctly, but the intuitive explanation he offered alongside the calculation was wrong.
The statistician Stephen Stigler gave a neat way to see that Newton’s intuitive argument had to be wrong, even without following it in detail. Nothing in Newton’s argument used the fact that the dice are fair. An argument that never uses the probability of a six would give the same ranking for any die. But the ranking does depend on that probability.
Suppose rolls remain independent but each has probability \(p\) of showing six. The exactly-\(r\) formula becomes:
\[\binom nr p^r(1-p)^{n-r}.\]For \(p=1/2\), the winning probabilities are approximately \(0.984375\), \(0.996826\), and \(0.999344\) for the three games. Their ordering reverses: now \(C\) is the most likely. Thus an argument claiming the fair-dice ordering without using the value \(p=1/6\) cannot establish the result.
Core idea: Even Newton’s intuition failed on this problem, while the calculation succeeded. A check that an argument actually uses every assumption the answer depends on can expose a flawed intuition.
Suppose we learn that event \(B\) occurred. Outcomes outside \(B\) are no longer compatible with the evidence. Among outcomes inside \(B\), the ones where \(A\) also occurs form \(A\cap B\).
For \(P(B)>0\):
\[\boxed{P(A\mid B)=\frac{P(A\cap B)}{P(B)}.}\]Read this as “the probability of \(A\) given \(B\).” The event after the conditioning bar is the information being treated as known.
The numerator retains the probability mass consistent with both events. Dividing by \(P(B)\) rescales the total probability mass inside \(B\) to 1.
In the equal-mass version of the diagram, \(B\) contains four of the nine pebbles and \(A\cap B\) contains one. Before conditioning, \(P(A\cap B)=1/9\) and \(P(B)=4/9\). After conditioning:
\[P(A\mid B)=\frac{1/9}{4/9}=\frac14.\]With unequal outcome probabilities, sum the masses instead of simply counting pebbles. Conditioning preserves the relative probabilities of the surviving outcomes.
Instead of performing the experiment once, imagine repeating it many times under the same conditions. In this frequentist view, a probability is the long-run fraction of repetitions in which an event occurs. For example, if a coin lands heads 612 times in 1000 tosses, \(P(\text{Heads})\) is about \(612/1000\).
To find \(P(A\mid B)\) from this point of view:
That fraction is approximately
\[\frac{\#\{\text{repetitions with }A\text{ and }B\}}{\#\{\text{repetitions with }B\}} =\frac{\#(A\cap B)/\#\text{total}}{\#B/\#\text{total}} \approx\frac{P(A\cap B)}{P(B)}.\]Restricting attention to the repetitions where \(B\) happened plays the role of removing pebbles outside \(B\). Dividing by the number of circled repetitions plays the role of renormalizing.
Roll a fair die. Let \(A=\{4,5,6\}\) and \(B=\{2,4,6\}\). Then:
\[P(A\mid B)=\frac{P(\{4,6\})}{P(\{2,4,6\})} =\frac{2/6}{3/6}=\frac23.\]Originally, \(P(A)=1/2\). The evidence changes the relevant possibilities to \(2,4,6\), of which two satisfy \(A\).
Core idea: Conditional probability changes the denominator to the probability of the evidence, not to the probability of the event being investigated.
The condition \(P(B)>0\) is essential. The elementary ratio does not define \(P(A\mid B)\) when \(P(B)=0\). Conditioning on probability-zero information requires additional machinery beyond this definition.
The conditioning bar is not a set operation. \(P(A\mid B)\) is a probability under specified information; it does not refer to an event called “\(A\mid B\).”
For fixed \(B\) with \(P(B)>0\), define \(Q(A)=P(A\mid B)\). This is itself a probability function:
\[Q(S)=\frac{P(S\cap B)}{P(B)}=1,\qquad Q(\varnothing)=0.\]If \(A_1,A_2,\ldots\) are disjoint, their intersections with \(B\) are disjoint, so:
\[Q\left(\bigcup_i A_i\right) =\frac{\sum_i P(A_i\cap B)}{P(B)} =\sum_i Q(A_i).\]Consequently, ordinary probability rules apply while keeping the evidence fixed. In particular:
\[P(A^c\mid B)=1-P(A\mid B),\] \[P(A\cup C\mid B)=P(A\mid B)+P(C\mid B)-P(A\cap C\mid B).\]Also, \(P(B\mid B)=1\), and \(P(A\mid B)=1\) whenever \(B\subseteq A\).
For \(P(B)>0\):
\[A\text{ and }B\text{ independent} \quad\Longleftrightarrow\quad P(A\mid B)=P(A).\]Indeed, substituting \(P(A\cap B)=P(A)P(B)\) into the definition cancels \(P(B)\). Conversely, multiply the no-update equation by \(P(B)\) to recover independence.
If \(P(A)>0\) as well, independence also gives \(P(B\mid A)=P(B)\). The product definition remains valid even when a conditional ratio would be undefined.
Rearranging the conditional-probability definition gives:
\[\boxed{P(A\cap B)=P(B)P(A\mid B),\qquad P(B)>0.}\]If \(P(A)>0\), we can also write:
\[P(A\cap B)=P(A)P(B\mid A).\]These are general multiplication rules. Independence allows the conditional factor to be replaced by its unconditional probability; without independence, the conditional factor must remain.
Draw two cards in order from a uniformly shuffled standard deck. Let \(H_1\) and \(H_2\) denote a heart on the first and second draws.
The first draw is a heart with probability \(13/52\). Given a first heart, twelve hearts remain among 51 cards:
\[P(H_1\cap H_2)=P(H_1)P(H_2\mid H_1) =\frac{13}{52}\frac{12}{51}=\frac1{17}.\]The unconditional probability of a heart on the second draw is still \(13/52=1/4\) by symmetry. But \(12/51\ne1/4\), so the two heart events are dependent.
With replacement and independent draws, the second factor would be \(13/52\), giving \(1/16\) instead.
Core idea: The first result changes the composition of the remaining deck. The multiplication rule accounts for that change through a conditional probability.
Applying the two-event rule repeatedly gives the probability that all of \(A_1,\ldots,A_n\) occur:
provided \(P(A_1\cap\cdots\cap A_{n-1})>0\).
Each new factor conditions on all the events already included. For example, with three events:
\[P(A\cap B\cap C)=P(A)P(B\mid A)P(C\mid A\cap B).\]The events can be introduced in any order, since the intersection does not depend on the order. There are \(n!\) orders, so this is really \(n!\) versions of the theorem. The left side stays the same, but the conditioning events on the right change with the order chosen. For some problems one order gives easy factors while another gives hard ones, so it is worth choosing the order deliberately.
If the events are independent, every conditional factor equals the unconditional probability, and the rule reduces to \(P(A_1)P(A_2)\cdots P(A_n)\).
When \(P(A)>0\) and \(P(B)>0\), the two multiplication rules describe the same intersection:
\[P(A\mid B)P(B)=P(A\cap B)=P(B\mid A)P(A).\]Divide by \(P(B)\):
Bayes’ rule is useful when the conditional probability in one direction is easier to calculate than the one in the other direction.
For a hypothesis \(H\) and evidence \(E\):
\[P(H\mid E)=\frac{P(E\mid H)P(H)}{P(E)}.\]| Quantity | Interpretation |
|---|---|
| Prior, \(P(H)\) | Probability assigned before incorporating evidence \(E\) |
| Likelihood, \(P(E\mid H)\) | Probability of observing the evidence if \(H\) holds |
| Evidence probability, \(P(E)\) | Overall probability of observing \(E\) |
| Posterior, \(P(H\mid E)\) | Updated probability after incorporating \(E\) |
A high likelihood does not by itself imply a high posterior. The prior and the overall probability of the evidence also matter.
Incorrect: \(P(A\mid B)=P(B\mid A)\) because both concern \(A\) and \(B\) occurring.
Correction: Both use the same numerator \(P(A\cap B)\), but divide by different probabilities. In general:
\[P(A\mid B)=\frac{P(A\cap B)}{P(B)},\qquad P(B\mid A)=\frac{P(A\cap B)}{P(A)}.\]Draw two cards without replacement. Let \(A\) mean the first card is a heart, and \(B\) mean the second card is red.
Find the easier direction: Given a first heart, 25 red cards remain among 51 cards:
\[P(B\mid A)=\frac{25}{51}.\]Find the unconditional probabilities: The first card is a heart with probability \(1/4\). Before either draw is observed, the second card is equally likely to be any of the 52 cards, so \(P(B)=1/2\).
Reverse the conditioning:
\[P(A\mid B)=\frac{(25/51)(1/4)}{1/2}=\frac{25}{102}.\]The two directions differ: \(25/102\) versus \(25/51\). As a check, their common intersection probability is:
\[P(A\cap B)=\frac14\frac{25}{51}=\frac{25}{204}.\]Core idea: Information about the second draw can update the probability of the first. Conditioning concerns information, not a causal influence traveling backward in time.
Consider a family with two children. Use an idealized model in which each child is independently a girl or a boy with equal probability. List the elder child first, so the four equally likely outcomes are:
\[S=\{GG,GB,BG,BB\}.\]Let \(F\) be the event that both children are girls.
Let \(E=\{GG,GB,BG\}\). Then \(F\cap E=F\), and:
\[P(F\mid E)=\frac{1/4}{3/4}=\frac13.\]The surviving family types are three equally likely outcomes, one of which is \(GG\).
Let \(E_1=\{GG,GB\}\). Then:
\[P(F\mid E_1)=\frac{1/4}{1/2}=\frac12.\]Here the evidence designates a particular child, leaving two equally likely possibilities for the younger child.
Core idea: Both descriptions guarantee a girl, but they define different events. Conditional probabilities depend on the precise information and how it was obtained. Observing a randomly selected child is another experiment; it should not automatically be treated as conditioning on “at least one girl.”
Try these before reading the answers.
Their intersection is empty, so they are disjoint. But:
\[P(A\cap B)=0\ne\frac26\frac26=\frac19.\]They are not independent. Learning that \(A\) occurred rules out \(B\).
Independence also holds for complements:
\[P(A\cap B^c)=0.30(0.60)=0.18,\] \[P(A^c\cap B^c)=0.70(0.60)=0.42.\]The union probability is:
\[P(A\cup B)=0.30+0.40-0.30(0.40)=0.58.\]It also equals \(1-0.42\), by taking the complement of neither event occurring.
All three events have probability \(1/2\). Each pairwise intersection is \(\{HH\}\), so its probability is \(1/4=(1/2)(1/2)\).
The triple intersection is also \(\{HH\}\), with probability \(1/4\), rather than \(1/8\). Thus the pairwise checks pass but the triple condition fails.
Exactly two sixes:
\[P(\text{exactly two})=\frac{\binom42 5^2}{6^4} =\frac{150}{1296}=\frac{25}{216}\approx0.115741.\]For at least two, subtract the disjoint zero-six and one-six cases:
\[P(\text{at least two})=1-\frac{5^4+4\cdot5^3}{6^4} =\frac{171}{1296}=\frac{19}{144}\approx0.131944.\]Failing to obtain at least two sixes means obtaining either zero or one. For exactly one, choose its position in twelve ways. Each of the other eleven dice has five allowed non-six results, giving \(12\cdot5^{11}\) outcomes. The two complement cases are disjoint, so their counts add.
Keeping the evidence \(B\) fixed:
\[P(A^c\mid B)=1-0.20=0.80.\]The events are dependent, since \(P(A)P(B)=0.20\ne0.10=P(A\cap B)\).
Without replacement:
\[P(\text{two aces})=\frac4{52}\frac3{51}=\frac1{221}\approx0.004525.\]With replacement and independent draws:
\[P(\text{two aces})=\left(\frac4{52}\right)^2=\frac1{169}\approx0.005917.\]Removing a first ace decreases the proportion of aces available for the second draw.
The joint probability is \(P(H\cap E)=0.20(0.60)=0.12\). The likelihood \(0.60\) and the posterior \(0.40\) answer different questions.
Given at least one boy, the surviving outcomes are \(BB,BG,GB\), each equally likely. Thus the probability of two boys is \(1/3\).
Given that the younger child is a boy, only \(BB,GB\) remain. The probability of two boys is \(1/2\). The difference comes from conditioning on different events.
| Concept | Essential fact |
|---|---|
| Independence | \(P(A\cap B)=P(A)P(B)\) |
| Disjointness | \(A\cap B=\varnothing\); disjoint positive-probability events are dependent |
| Complements of independent events | Complementing either or both preserves independence |
| Mutual independence | The intersection product rule must hold for every subset of events |
| Exactly \(r\) sixes in \(n\) fair independent rolls | \(\binom nr5^{n-r}/6^n\) |
| Newton–Pepys ranking | One six in six rolls is more likely than two in twelve or three in eighteen |
| Stigler’s check | Newton’s intuition ignored fairness, but with \(p=1/2\) the ranking reverses |
| Conditional probability | \(P(A\mid B)=P(A\cap B)/P(B)\), with \(P(B)>0\) |
| Conditioning intuition | Remove outcomes outside the evidence, then renormalize |
| Frequentist intuition | Among many repetitions where \(B\) occurred, the fraction where \(A\) occurred |
| Conditional complement | \(P(A^c\mid B)=1-P(A\mid B)\) |
| Independence as no update | For \(P(B)>0\), \(P(A\mid B)=P(A)\) |
| Multiplication rule | \(P(A\cap B)=P(B)P(A\mid B)\) |
| Chain rule | \(P(A_1\cdots A_n)=P(A_1)P(A_2\mid A_1)\cdots P(A_n\mid A_1,\ldots,A_{n-1})\), in any of \(n!\) orders |
| Bayes’ rule | \(P(A\mid B)=P(B\mid A)P(A)/P(B)\) |
| Direction of conditioning | \(P(A\mid B)\) and \(P(B\mid A)\) generally differ |
Events whose intersection probability equals the product of their individual probabilities. For positive-probability evidence, conditioning on one leaves the probability of the other unchanged.
Independence of every pair in a collection of events. This does not guarantee independence of the collection as a whole.
The intersection product rule holding for every subset of two or more events in a collection.
A probability calculated with specified evidence treated as known, using the mass of the intersection divided by the mass of the evidence.
Rescaling surviving probability masses so they total 1. When conditioning on \(B\), each surviving mass is divided by \(P(B)\).
The probability of a hypothesis before incorporating the specified new evidence. It can already reflect other background information.
The probability of the observed evidence given a hypothesis, \(P(E\mid H)\). It is not the probability of the hypothesis given the evidence.
The updated probability of a hypothesis after conditioning on evidence, \(P(H\mid E)\).
A relation between the two directions of conditioning, obtained by writing the same intersection probability in two ways.
Conditional probabilities depend on the exact evidence being used. The law of total probability combines simpler conditional calculations across disjoint cases, supplying the denominator needed in Bayes’ rule. Conditional independence allows multiplication within a specified context, but mixing contexts or selecting outcomes can create dependence that was absent within the original model.
| Symbol | Meaning |
|---|---|
| \(P(A\mid B)\) | Probability of \(A\) given \(B\), with \(P(B)>0\) |
| \(A_1,\ldots,A_n\) | A partition of \(S\): disjoint cases covering the whole sample space |
| \(P(B\mid A_i)P(A_i)\) | Joint probability \(P(B\cap A_i)\) |
| \(\sum_i P(B\mid A_i)P(A_i)\) | Law of total probability for \(P(B)\) |
| \(H_i\) | One of several mutually exclusive, exhaustive hypotheses |
| \(E\) | Evidence being conditioned on |
| \(P(A\mid B,E)\) | Probability of \(A\) given both \(B\) and \(E\); commas mean intersections |
| \(D\), \(T\) | Disease and positive test result in a hypothetical test model |
| \(P(T\mid D)\) | Sensitivity: true-positive rate |
| \(P(T^c\mid D^c)\) | Specificity: true-negative rate |
| \(P(A\cap B\mid E)=P(A\mid E)P(B\mid E)\) | Conditional independence of \(A\) and \(B\) given \(E\) |
After studying these notes, you should be able to:
Draw two cards without replacement from a uniformly shuffled standard deck. Let \(F\) be the event that both cards are aces.
Because these questions concern the two-card hand rather than its draw order, use unordered hands as outcomes. There are:
\[\binom{52}{2}=1326\]equally likely hands, of which \(\binom42=6\) contain two aces.
Let \(E\) be the event that the hand contains at least one ace. Count it using two disjoint cases:
| Case | Count | Reason |
|---|---|---|
| Exactly one ace | \(4\cdot48=192\) | Choose one ace and one non-ace |
| Two aces | \(\binom42=6\) | Choose two of the four aces |
Thus \(\lvert E\rvert=198\). Since \(F\subseteq E\):
\[\boxed{P(F\mid E)=\frac{6/1326}{198/1326}=\frac6{198}=\frac1{33}.}\]An equivalent denominator is \(\binom{52}{2}-\binom{48}{2}\), subtracting hands with no aces.
Let \(E_s\) mean that the ace of spades is one of the two cards. Fix that card and choose its companion from the other 51 cards. There are 51 such hands, and three have another ace:
\[\boxed{P(F\mid E_s)=\frac3{51}=\frac1{17}.}\]The evidence identifies a particular card. The remaining card is uniformly distributed among the 51 other cards.
This evidence refers to draw order, so now use ordered outcomes. Given that the first card is an ace, three aces remain among 51 cards:
\[P(\text{two aces}\mid\text{first card is an ace})=\frac3{51}=\frac1{17}.\]Cases 2 and 3 happen to give the same answer, but they are different conditioning events. Neither is equivalent to merely knowing that at least one card is an ace.
Incorrect: Knowing that there is an ace lets us remove it and treat the other card as uniform among 51 cards, giving \(1/17\) in every case.
Correction: “At least one ace” does not designate which card was identified. Among its 198 compatible hands, only six contain two aces. The particular-card evidence and the at-least-one evidence select different collections of hands.
Core idea: A more specific conditioning event can change the probability of another event. The difference comes from the outcomes compatible with the evidence, not from the words “an ace” alone.
Events \(A_1,\ldots,A_n\) form a partition of \(S\) if:
\[A_i\cap A_j=\varnothing\quad(i\ne j),\qquad \bigcup_{i=1}^{n}A_i=S.\]Each outcome belongs to exactly one case. To use conditional probabilities \(P(B\mid A_i)\), assume each case has positive probability.
For any event \(B\), the pieces \(B\cap A_i\) are disjoint and together cover \(B\):
\[B=\bigcup_{i=1}^{n}(B\cap A_i).\]By additivity and the multiplication rule:
\[P(B)=\sum_{i=1}^{n}P(B\cap A_i) =\sum_{i=1}^{n}P(B\mid A_i)P(A_i).\]This is the law of total probability, abbreviated LOTP.
The weight \(P(A_i)\) is the chance of being in case \(i\); \(P(B\mid A_i)\) is the chance of \(B\) within that case. Their product is the contribution of that case to the overall probability.
Core idea: The unconditional probability is a weighted average of conditional probabilities. The weights sum to 1; they need not be equal.
For \(0<P(A)<1\), the cases \(A\) and \(A^c\) form a partition:
\[\boxed{P(B)=P(B\mid A)P(A)+P(B\mid A^c)P(A^c).}\]Disjoint and exhaustive are both required. Overlapping cases double-count some outcomes; cases that do not cover \(S\) omit others. Adding \(P(B\mid A_i)\) without multiplying by \(P(A_i)\) also gives the wrong weighting.
If a case has probability zero, its joint contribution is zero. Omit it rather than treating an undefined conditional probability as a number to multiply by zero.
Let \(H_1,\ldots,H_n\) be a partition with positive prior probabilities, and suppose \(P(E)>0\). Bayes’ rule gives:
\[P(H_j\mid E)=\frac{P(E\mid H_j)P(H_j)}{P(E)}.\]Use LOTP to calculate the denominator:
\[\boxed{P(H_j\mid E)= \frac{P(E\mid H_j)P(H_j)} {\sum_{i=1}^{n}P(E\mid H_i)P(H_i)}.}\]The numerator is the probability of hypothesis \(j\) together with the evidence. The denominator is the total probability of that evidence across every possible hypothesis. Dividing assigns the fraction of the evidence probability attributable to hypothesis \(j\).
Choose once between a fair coin and a biased coin, each with probability \(1/2\). The biased coin lands heads with probability \(3/4\). Toss the selected coin three times, independently given which coin was selected, and observe \(HHH\).
Let \(F\) mean the coin is fair and \(E\) mean three heads. The likelihoods are:
\[P(E\mid F)=\left(\frac12\right)^3=\frac18, \qquad P(E\mid F^c)=\left(\frac34\right)^3=\frac{27}{64}.\]The total evidence probability is:
\[P(E)=\frac18\frac12+\frac{27}{64}\frac12 =\frac8{128}+\frac{27}{128}=\frac{35}{128}.\]Hence:
\[\boxed{P(F\mid E)=\frac{(1/8)(1/2)}{35/128} =\frac8{35}\approx0.228571.}\]The biased coin has posterior probability \(27/35\). Three heads are possible under either coin, but are more likely under the biased coin, so the observation shifts probability toward that coin.
Core idea: Multiply within each hypothesis, add across the mutually exclusive hypotheses, and normalize to update their probabilities.
Incorrect: We observed \(E\), so substitute \(P(E)=1\) into Bayes’ rule.
Correction: Observing \(E\) makes \(P(E\mid E)=1\). The denominator \(P(E)\) in Bayes’ rule is the probability of the evidence under the original model, before conditioning on it.
Consider a hypothetical test model. Let \(D\) be the event that a person has a condition and \(T\) the event of a positive result. Assume the person is drawn from a population with condition prevalence \(p=P(D)\).
“95% accurate” is ambiguous unless the relevant conditional probabilities are stated. In this model, take:
\[P(T\mid D)=0.95,\qquad P(T^c\mid D^c)=0.95.\]Thus the false-positive rate is \(P(T\mid D^c)=0.05\). These are assumptions for an illustrative probability problem.
The desired probability after a positive result is \(P(D\mid T)\), not \(P(T\mid D)\). By Bayes’ rule and LOTP:
\[\boxed{P(D\mid T)= \frac{0.95p}{0.95p+0.05(1-p)}.}\]The denominator includes both ways of getting a positive result: a true positive and a false positive.
Suppose 1% of patients similar to this one have the condition, so \(p=0.01\):
\[P(T)=0.95(0.01)+0.05(0.99)=0.0095+0.0495=0.059,\] \[P(D\mid T)=\frac{0.0095}{0.059} =\frac{19}{118}\approx0.161017.\]The probability rises from 1% before the result to about 16.10% afterward. The result is informative even though the posterior is much lower than the sensitivity.
For 10,000 people under these proportions:
| Group | Expected positive results | Expected negative results | Total |
|---|---|---|---|
| Condition present | 95 | 5 | 100 |
| Condition absent | 495 | 9405 | 9900 |
| Total | 590 | 9410 | 10,000 |
Among positive results, the expected fraction with the condition is \(95/590=19/118\). A small false-positive rate applied to a large condition-free group can produce more positives than a high true-positive rate applied to a small group.
A rough frequency check. Imagine 1000 such patients. About 10 have the condition, and nearly all of them test positive. About 990 do not, and 5% of those, roughly 50, also test positive. So about 10 of roughly 60 positive results come from patients with the condition, a ratio near \(1/6\approx0.16\). The rarity of the condition competes with the rarity of a test error; intuition that focuses only on the 95% ignores the 1%.
If the patient already has symptoms, the starting probability is no longer 1%. The method is unchanged: start from the probability that reflects the evidence already available, and update it with Bayes’ rule.
Keeping the same test assumptions but changing the prevalence to \(1/1000\) gives:
\[P(D\mid T)=\frac{0.95/1000}{0.95/1000+0.05(999/1000)} =\frac{19}{1018}\approx0.018664.\]The sensitivity and specificity are unchanged, but the posterior is now about 1.87%. The prior prevalence affects the relative contributions of true and false positives.
Core idea: A conditional probability about how evidence is generated cannot be read directly as a probability about its underlying cause. The base rate supplies essential information.
A small probability of evidence given innocence, \(P(E\mid I)\), does not equal a small probability of innocence given the evidence, \(P(I\mid E)\). Bayes’ rule also requires the prior probabilities and the probability of the evidence under alternatives. Confusing these directions is called the prosecutor’s fallacy. The name is unfair to prosecutors: defense attorneys, doctors, and everyone else make the same error.
Sally Clark, a British woman, lost two infants to sudden, unexplained deaths and was convicted of murdering them. The central statistical testimony claimed that, for an innocent family, the chance of one such death is about \(1/8500\). Multiplying for two deaths gave
\[\left(\frac1{8500}\right)^2\approx\frac1{72{,}000{,}000},\]reported as about 1 in 73 million. The argument contains two separate errors.
The prior \(P(I)\) is extremely close to 1, because the vast majority of mothers do not murder their children. The denominator must also account for the probability of two deaths under the alternative explanation, which is itself extremely small. Treating \(1/73\) million as the probability of innocence ignores both.
The conviction was later overturned, but only after she had spent years in prison.
The probability \(P(A)\) before observing evidence is the prior probability of \(A\). After observing \(B\), the updated probability \(P(A\mid B)\) is the posterior probability.
Incorrect: The problem says that \(A\) occurred, so write \(P(A)=1\).
Correction: What is true is \(P(A\mid A)=1\). The symbol \(P(A)\) still means the probability of \(A\) before that information. Put observed evidence to the right of the conditioning bar, and keep \(P(A)\) for the prior.
For a fixed event \(E\) with \(P(E)>0\), the function \(Q(B)=P(B\mid E)\) obeys the probability axioms. Consequently, LOTP and Bayes’ rule can be applied inside that conditional model.
For a partition \(A_1,\ldots,A_n\), retaining positive-probability cases within \(E\):
\[\boxed{P(B\mid E)=\sum_{i=1}^{n} P(B\mid A_i,E)P(A_i\mid E).}\]The weights are now \(P(A_i\mid E)\), not the original \(P(A_i)\). The evidence can change the distribution of the cases themselves.
When \(P(A\cap E)>0\) and \(P(B\cap E)>0\):
\[\boxed{P(A\mid B,E)= \frac{P(B\mid A,E)P(A\mid E)}{P(B\mid E)}.}\]The same background evidence \(E\) appears in every probability. Commas to the right of the bar denote joint evidence, so \(P(B\mid A,E)=P(B\mid A\cap E)\).
Bayes’ rule is coherent: updating on two pieces of evidence at once gives the same answer as updating on one, then using that posterior as the new prior and updating on the other.
Suppose clues \(B_1\) and \(B_2\) arrive with \(P(B_1\cap B_2)>0\). After the first clue, the current probability function is \(Q(\cdot)=P(\cdot\mid B_1)\). Updating \(Q\) on the second clue gives:
\[Q(A\mid B_2)=\frac{Q(A\cap B_2)}{Q(B_2)} =\frac{P(A\cap B_1\cap B_2)/P(B_1)}{P(B_1\cap B_2)/P(B_1)} =P(A\mid B_1\cap B_2).\]This is exactly the all-at-once update. Since \(B_1\cap B_2=B_2\cap B_1\), the order of the clues does not matter either. Whether the evidence comes all at once, one piece at a time, or in any order, the result is the probability of \(A\) given all the evidence.
Core idea: Once a calculation is conditional on a context, both the case probabilities and within-case probabilities must use that context. Sequential updating is the same principle applied repeatedly.
For \(P(E)>0\), events \(A\) and \(B\) are conditionally independent given \(E\) if:
\[\boxed{P(A\cap B\mid E)=P(A\mid E)P(B\mid E).}\]If also \(P(B\cap E)>0\), this is equivalent to:
\[P(A\mid B,E)=P(A\mid E).\]Given the context \(E\), learning \(B\) supplies no further information that changes the probability of \(A\). This statement is about the probability model after conditioning, not necessarily about the original model.
Independence does not imply conditional independence. Conditional independence does not imply independence. Independence given \(E\) also need not imply independence given \(E^c\).
Return to choosing once between the fair coin and the \(3/4\)-heads coin. Let \(A\) and \(B\) be heads on the first and second tosses.
Given the fair coin:
\[P(A\cap B\mid F)=\frac14=P(A\mid F)P(B\mid F).\]Given the biased coin:
\[P(A\cap B\mid F^c)=\frac9{16}=P(A\mid F^c)P(B\mid F^c).\]Thus the tosses are independent within each coin type. Without knowing the type:
\[P(A)=P(B)=\frac12\frac12+\frac34\frac12=\frac58,\] \[P(A\cap B)=\frac14\frac12+\frac9{16}\frac12=\frac{13}{32}.\]But:
\[\frac{13}{32}=\frac{26}{64}\ne\frac{25}{64} =\left(\frac58\right)^2.\]Indeed:
\[P(B\mid A)=\frac{13/32}{5/8}=\frac{13}{20}=0.65>\frac58.\]A first head supplies evidence that the selected coin is biased, which in turn raises the probability of a second head.
Core idea: Multiplying within each known case is valid; averaging over an unknown shared case can introduce dependence. Choosing a fresh coin independently before every toss would describe a different experiment.
Suppose repeated game results are independent once an opponent’s strength is known. If the opponent’s strength is unknown, an observed loss can make a strong opponent more plausible, changing the probability of losing the next game. Conditional independence within each strength category does not imply independence after those categories are mixed.
This is the same structure as the shared-coin example: one unknown factor affects several outcomes, and observing one outcome supplies information about that factor.
Independent events need not be conditionally independent. This is common whenever one outcome has several possible causes.
Suppose a fire alarm sounds exactly when there is a fire or someone is making popcorn, and nothing else sets it off. Let \(F\) mean there is a fire, \(C\) mean someone is making popcorn, and \(A=F\cup C\) mean the alarm sounds. Assume \(F\) and \(C\) are independent, with positive probabilities less than 1.
Given that the alarm sounds and no one is making popcorn, the only remaining explanation is a fire:
\[P(F\mid A,C^c)=1.\]Given only that the alarm sounds, a fire is not certain, because popcorn could explain it:
\[P(F\mid A)=\frac{P(F)}{P(F\cup C)}<1.\]So, given \(A\), learning whether \(C\) occurred changes the probability of \(F\). The events \(F\) and \(C\) are independent, but not conditionally independent given \(A\).
For illustration, take \(P(F)=0.1\) and \(P(C)=0.3\). Then \(P(A)=1-(0.9)(0.7)=0.37\) and:
| Information | Probability of fire |
|---|---|
| None | \(0.1\) |
| Alarm sounds | \(0.1/0.37\approx0.270\) |
| Alarm sounds and popcorn is being made | \(0.1\) |
| Alarm sounds and no popcorn | \(1\) |
Learning that popcorn is being made “explains away” the alarm: it accounts for the alarm, so the probability of fire falls back to 0.1. Learning that there is no popcorn removes the other explanation, so fire becomes certain.
Toss two independent fair coins. Let \(A\) mean first heads, \(B\) mean second heads, and \(E=A\cup B\) mean at least one head.
Before conditioning, \(A\) and \(B\) are independent. Given \(E\), the three equally likely surviving outcomes are:
\[HH,\quad HT,\quad TH.\]Therefore:
\[P(A\mid E)=P(B\mid E)=\frac23,\]but:
\[P(A\cap B\mid E)=\frac13\ne\frac49 =P(A\mid E)P(B\mid E).\]Once at least one head is known, learning that the first toss is tails forces the second to be heads:
\[P(B\mid A^c,E)=1.\]The independent events have become dependent under the selected evidence.
Incorrect: Independence is a permanent property, so conditioning cannot change it.
Correction: Independence refers to a particular probability function. Conditioning changes that function. Always specify the context in which multiplication is justified.
Core idea: Restricting the sample space can remove combinations that originally made independence possible. Conditional independence must be checked using probabilities with the same evidence throughout.
Try these before reading the answers.
There are \(4\cdot48+\binom42=198\) hands containing at least one king, of which six have two kings:
\[P(\text{two kings}\mid\text{at least one king})=\frac6{198}=\frac1{33}.\]Given the king of hearts, its companion is one of 51 other cards, of which three are kings. The probability is \(3/51=1/17\).
By LOTP:
\[P(R)=0.80(0.30)+0.20(0.70)=0.38.\]By Bayes’ rule:
\[P(A\mid R)=\frac{0.80(0.30)}{0.38}=\frac{12}{19}\approx0.631579.\]The more red-heavy box becomes more likely after observing red.
The events overlap: a hand may contain both an ace and a heart. They also fail to cover the sample space: a hand may contain neither. A valid partition could instead use the four combinations of containing or not containing an ace and containing or not containing a heart.
The likelihoods are \(1/4\) and \(9/16\). With equal priors:
\[P(F\mid HH)=\frac{(1/4)(1/2)}{(1/4)(1/2)+(9/16)(1/2)} =\frac4{13}\approx0.307692.\]The false-positive rate is \(1-0.95=0.05\). Thus:
\[P(T)=0.90(0.02)+0.05(0.98)=0.067,\] \[P(D\mid T)=\frac{0.018}{0.067}=\frac{18}{67}\approx0.268657.\]The posterior is about 26.87%, rather than the 90% sensitivity.
The false-negative rate is \(0.10\). Therefore:
\[P(T^c)=0.10(0.02)+0.95(0.98)=0.933,\] \[P(D\mid T^c)=\frac{0.10(0.02)}{0.933} =\frac2{933}\approx0.002144.\]This is about 0.2144%. The positive and negative evidence probabilities sum to 1, but their corresponding condition posteriors do not have to do so: they condition on different events.
These calculations apply within \(E\). They do not determine whether \(A\) and \(B\) are independent without conditioning.
The first head favors the biased coin, increasing the probability of heads on the second toss. The tosses are independent given coin type, but not after mixing the two types.
Given \(A\) and \(E\), only \(HH\) and \(HT\) survive, with equal probabilities, so:
\[P(B\mid A,E)=\frac12.\]Given only \(E\), the three outcomes \(HH,HT,TH\) survive, giving \(P(B\mid E)=2/3\). The difference shows that \(A\) and \(B\) are not conditionally independent given \(E\).
Let \(R\) be rain, \(S\) the sprinkler, and \(W=R\cup S\) a wet lawn. Then \(P(W)=1-(0.8)(0.5)=0.6\), and since \(R\subseteq W\):
\[P(R\mid W)=\frac{0.2}{0.6}=\frac13.\]Given \(W\) and \(S\), the lawn is wet regardless of rain, so the wet lawn adds nothing: \(P(R\mid W,S)=P(R\mid S)=P(R)=0.2\). Learning that the sprinkler ran lowers the probability of rain from \(1/3\) to \(0.2\), so \(R\) and \(S\) are dependent given \(W\).
| Concept | Essential fact |
|---|---|
| Exact evidence | At least one ace gives \(1/33\) for two aces; a specified ace gives \(1/17\) |
| Partition | Pairwise disjoint cases whose union is \(S\) |
| LOTP | \(P(B)=\sum_i P(B\mid A_i)P(A_i)\) |
| Two-case LOTP | Split into \(A\) and \(A^c\), with both conditional probabilities defined |
| Partition form of Bayes | Posterior equals one likelihood-times-prior contribution divided by their total |
| Sensitivity | \(P(T\mid D)\) |
| Specificity | \(P(T^c\mid D^c)\) |
| False-positive rate | \(P(T\mid D^c)=1-\text{specificity}\) |
| Positive-result posterior | \(P(D\mid T)\) depends on prevalence as well as test rates |
| 1% prevalence, 95% accurate test | \(P(D\mid T)\approx0.16\): about 10 true positives against 50 false ones per 1000 |
| Prosecutor’s fallacy | \(P(E\mid I)\) is not \(P(I\mid E)\); Sally Clark also assumed unjustified independence |
| Prior and posterior | \(P(A)\) before evidence, \(P(A\mid B)\) after; observing \(A\) gives \(P(A\mid A)=1\), not \(P(A)=1\) |
| Coherence of Bayes | Updating all at once or one clue at a time, in any order, gives \(P(A\mid B_1,B_2)\) |
| Background evidence | Keep the same context to the right of the bar throughout a calculation |
| Conditional independence | \(P(A\cap B\mid E)=P(A\mid E)P(B\mid E)\) |
| Unknown shared factor | Independence within each case can become dependence after mixing cases |
| Selected evidence | Unconditional independence can disappear after conditioning |
| Explaining away | Independent causes \(F,C\) of an alarm become dependent given the alarm |
Disjoint events covering the whole sample space, so every outcome belongs to exactly one case.
A formula combining within-case conditional probabilities with the probabilities of the cases to recover an overall probability.
The prior frequency or probability of a category before incorporating the specified new evidence. In the test model, it is condition prevalence.
The probability of a positive test result given that the condition is present, \(P(T\mid D)\).
The probability of a negative test result given that the condition is absent, \(P(T^c\mid D^c)\).
Confusing the probability of evidence under innocence with the probability of innocence given that evidence.
The probability of an event before observing specified evidence, \(P(A)\), and after observing it, \(P(A\mid B)\).
The dependence created between independent causes when conditioning on an effect they can each produce: ruling out one cause raises the probability of the others.
Independence under a specified conditional probability model. It does not establish independence outside that context.
A probability model formed by combining different case-specific models with weights given by their case probabilities.
Information depends on how it is generated. In Monty Hall, a knowledgeable host’s deliberate goat reveal differs from opening a door at random. In Simpson’s paradox, comparisons within every subgroup can reverse after aggregation because the groups receive different weights. Both problems become clear when the conditioning events and the probabilities of the cases are written explicitly.
| Symbol | Meaning |
|---|---|
| \(C_i\) | Car is behind door \(i\) |
| \(M_j\) | Monty opens door \(j\) under the specified host protocol |
| \(W\) | The contestant wins by following the switching strategy |
| \(P(M_j\mid C_i)\) | Probability of the host’s action given the car location |
| \(N\) | Number of doors in a supplementary generalized game |
| \(S\) | Successful procedure in the Simpson example |
| \(H\), \(H^c\) | Heart procedure and Band-Aid removal |
| \(D_H\), \(D_N\) | Procedure performed by Dr. Hibbert and Dr. Nick |
| \(P(S\mid D_H,H)\) | Success rate for Dr. Hibbert within heart procedures |
| \(P(H\mid D_H)\) | Fraction of Dr. Hibbert’s procedures that are heart procedures |
| \(w\) | A common weight assigned to heart procedures in a standardized comparison |
After studying these notes, you should be able to:
Three closed doors hide one car and two goats. The contestant chooses a door. Monty opens one of the other doors, revealing a goat, and offers the contestant the chance to switch to the remaining closed door.
Use the standard model:
Label the contestant’s initial choice door 1. Let \(C_i\) mean that the car is behind door \(i\). Then:
\[P(C_1)=P(C_2)=P(C_3)=\frac13.\]Should the contestant stay with door 1 or switch?
The host protocol is part of the probability model. Knowing only that a door opened and a goat appeared does not specify how that evidence was generated. The host’s knowledge, choice rule, and decision to offer a switch affect the relevant probabilities.
Condition on the car location:
| Car location | Initial door 1 | Door Monty can open | Outcome if switching |
|---|---|---|---|
| Door 1 | Car | Door 2 or door 3 | Lose |
| Door 2 | Goat | Door 3 only | Win |
| Door 3 | Goat | Door 2 only | Win |
The initial choice is correct with probability \(1/3\). If it is wrong, Monty must reveal the other goat, leaving the car behind the remaining unchosen door.
By LOTP:
\[\begin{aligned} P(W) &=P(W\mid C_1)P(C_1)+P(W\mid C_2)P(C_2)+P(W\mid C_3)P(C_3)\\ &=0\cdot\frac13+1\cdot\frac13+1\cdot\frac13\\ &=\frac23. \end{aligned}\]
The two losing paths each have probability \((1/3)(1/2)=1/6\). The two winning paths each have probability \((1/3)(1)=1/3\). These four probabilities sum to 1.
Core idea: Switching exchanges the event “the original choice is correct” for its complement. Monty’s guaranteed goat reveal makes that exchange possible.
The overall \(2/3\) success rate does not depend on how Monty breaks a tie between two goats. If the original choice is wrong, he has no choice; if it is right, either goat reveal makes switching lose. Tie-breaking matters when conditioning on which particular door he opened, as the next calculation shows.
Suppose Monty opens door 2. The evidence is \(M_2\). Under the equal tie-breaking rule:
| Car location | \(P(C_i)\) | \(P(M_2\mid C_i)\) | \(P(C_i\cap M_2)\) |
|---|---|---|---|
| Door 1 | \(1/3\) | \(1/2\) | \(1/6\) |
| Door 2 | \(1/3\) | \(0\) | \(0\) |
| Door 3 | \(1/3\) | \(1\) | \(1/3\) |
By LOTP:
\[P(M_2)=\frac12\frac13+0\frac13+1\frac13=\frac12.\]Bayes’ rule gives:
\[P(C_1\mid M_2)=\frac{(1/2)(1/3)}{1/2}=\frac13,\] \[P(C_3\mid M_2)=\frac{1\cdot(1/3)}{1/2}=\frac23.\]Thus switching to door 3 wins with conditional probability \(2/3\). If Monty instead opens door 3, symmetry gives the same advantage for switching to door 2.
Incorrect: There are two closed doors and one car, so each door has probability \(1/2\).
Correction: Counting two possibilities does not establish equal probabilities. Monty’s action is more likely when the car is behind door 3 than when it is behind the originally chosen door 1. Bayes’ rule retains this difference in the posterior probabilities.
Core idea: Monty is not eliminating an arbitrary possibility. His choice is constrained by the hidden car location, so the reveal carries information.
Checking an extreme case helps. Suppose there are a million doors, you pick one, and Monty, who knows where the car is, opens 999,998 of the others, all goats, leaving one closed door besides yours. Almost no one would stay. Your first pick was almost certainly wrong, and Monty had to avoid the car when choosing which door to leave closed.
Yet the “two closed doors, so 50–50” argument applies just as well here as with three doors. Since it is clearly wrong for a million doors, it is wrong for three.
Supplementary generalization. With \(N\ge3\) doors, a uniformly placed car, and a host who always reveals \(N-2\) unchosen goats, the contestant’s first pick is a goat with probability \((N-1)/N\), and in every such case the surviving alternative holds the car:
\[P(\text{win by switching})=\frac{N-1}{N},\qquad P(\text{win by staying})=\frac1N.\]For a million doors, switching wins with probability \(999999/1000000\).
Suppose that, when the car is behind door 1, Monty opens door 2 with probability \(q\) and door 3 with probability \(1-q\). In other cases he still must reveal the only unchosen goat, and he always offers a switch.
After he opens door 2:
\[P(M_2)=q\frac13+1\frac13=\frac{1+q}{3},\] \[P(C_3\mid M_2)=\frac{1/3}{(1+q)/3}=\frac1{1+q}, \qquad P(C_1\mid M_2)=\frac q{1+q}.\]For \(q=1/2\), these are \(2/3\) and \(1/3\). For \(q=1\), this particular reveal leaves a \(1/2\)–\(1/2\) comparison. Yet the overall probability of winning by always switching is still \(2/3\).
For example, with \(q=1\), Monty opens door 3 only when the car is behind door 2. That reveal makes switching certain to win. The conditional rates average to:
\[P(W)=\frac12\frac23+1\frac13=\frac23.\]Now suppose the host does not know where the car is and chooses randomly between doors 2 and 3. Condition on the observation that he opened door 2 and it contained a goat. Call this evidence \(E_2\).
The likelihoods are:
\[P(E_2\mid C_1)=\frac12,\qquad P(E_2\mid C_2)=0,\qquad P(E_2\mid C_3)=\frac12.\]Unlike the knowledgeable host, the blind host is equally likely to generate this evidence under \(C_1\) and \(C_3\). Consequently:
\[P(E_2)=\frac13, \qquad P(C_1\mid E_2)=P(C_3\mid E_2)=\frac12.\]Switching and staying are equally likely to win under this conditional experiment.
Core idea: The visible outcome may look identical—a goat behind an opened door—but different selection mechanisms give different likelihoods and posteriors.
A comparison can reverse when data from different subgroups are combined. This phenomenon is called Simpson’s paradox.
Is it possible for one doctor to have a higher success rate than another for every type of surgery, yet a lower success rate overall? Consider Dr. Hibbert and Dr. Nick, the two doctors from The Simpsons, with made-up records of 100 procedures each. Heart procedures are harder than Band-Aid removals. The numbers are an illustration of aggregation, not evidence about actual clinical practice.
| Procedure | Dr. Hibbert: successes / total | Dr. Hibbert: success rate | Dr. Nick: successes / total | Dr. Nick: success rate |
|---|---|---|---|---|
| Heart | \(70/90\) | \(7/9\approx77.78\%\) | \(2/10\) | \(20\%\) |
| Band-Aid removal | \(10/10\) | \(100\%\) | \(81/90\) | \(90\%\) |
| All procedures | \(80/100\) | \(80\%\) | \(83/100\) | \(83\%\) |
Dr. Hibbert has the higher success rate in both procedure types, but Dr. Nick has the higher overall success rate.
Dr. Hibbert performs mostly difficult procedures. Dr. Nick performs mostly easy procedures. Their overall rates average over very different case mixes.
Core idea: An aggregate rate combines performance within groups and the proportions of cases belonging to those groups. A higher aggregate rate need not imply a higher rate within a comparable group.
Let \(S\) mean success and \(H\) mean a heart procedure. Let \(D_H\) mean Dr. Hibbert performed the procedure and \(D_N\) mean Dr. Nick performed it.
Keeping doctor identity as background evidence, LOTP gives:
\[\begin{aligned} P(S\mid D_H) ={}&P(S\mid D_H,H)P(H\mid D_H)\\ &+P(S\mid D_H,H^c)P(H^c\mid D_H). \end{aligned}\]For Dr. Hibbert:
\[P(S\mid D_H)=\frac79\frac9{10}+1\frac1{10} =\frac7{10}+\frac1{10}=0.80.\]For Dr. Nick:
\[P(S\mid D_N)=\frac15\frac1{10}+\frac9{10}\frac9{10} =\frac2{100}+\frac{81}{100}=0.83.\]The weights differ:
\[P(H\mid D_H)=0.90,\qquad P(H\mid D_N)=0.10.\]Dr. Hibbert’s lower within-doctor rate—the heart rate—receives much more weight than Dr. Nick’s heart rate. Dr. Nick’s easier-procedure rate receives most of his weight.
For an outcome \(A\), a group indicator \(B\), and a subgroup indicator \(C\), a reversal can have the form:
\[P(A\mid B,C)>P(A\mid B^c,C),\] \[P(A\mid B,C^c)>P(A\mid B^c,C^c),\]but:
\[P(A\mid B)<P(A\mid B^c).\]All conditioning events must have positive probability. The reversal is possible because the overall probabilities use different subgroup weights, \(P(C\mid B)\) and \(P(C\mid B^c)\).
The variable \(C\) is called a confounder: a variable related to both the group and the outcome, which should be controlled for. In the doctor example, \(A\) is success, \(B\) is treatment by Dr. Nick, and \(C\) is a heart procedure. Without conditioning on \(C\), learning that Dr. Nick operated tells us the procedure was probably an easy one, and that information, not his skill, raises his aggregate rate. Here the more relevant comparison is the one that conditions on the procedure type.
Aggregating records adds the successes and adds the totals. That is the “wrong” way to add fractions:
\[\frac13+\frac25\ne\frac{1+2}{3+5}=\frac38.\]The combined rate \((a+c)/(b+d)\) is not determined by \(a/b\) and \(c/d\) alone:
\[\frac{a+c}{b+d}=\frac{b}{b+d}\cdot\frac ab+\frac{d}{b+d}\cdot\frac cd,\]a weighted average whose weights depend on the group sizes \(b\) and \(d\). Two doctors with different case mixes are averaged with different weights, so being better within every group does not guarantee being better after aggregation.
A batting average is the fraction of at-bats that produce a hit. With illustrative numbers:
| Player | First half | Second half | Whole season |
|---|---|---|---|
| A | \(4/10=0.400\) | \(25/100=0.250\) | \(29/110\approx0.264\) |
| B | \(35/100=0.350\) | \(2/10=0.200\) | \(37/110\approx0.336\) |
Player A has the higher average in each half, but Player B has the higher average for the season. Most of A’s at-bats fall in the weaker second half, and most of B’s fall in the stronger first half.
Graduate admissions. A lawsuit against UC Berkeley alleged discrimination against women in graduate admissions, because the overall admission rate was higher for men. Within individual departments, there was no clear evidence of discrimination against women. Women applied more often to departments that were harder to get into, so aggregating across departments produced the apparent gap. The department is the confounder.
Each of four jars holds jelly beans of a liked flavor and a disliked flavor, and you draw one at random from a jar of your choice. With illustrative counts of liked beans out of total beans:
| Jar | Liked / total | Rate |
|---|---|---|
| \(A_1\) | \(2/3\) | \(0.667\) |
| \(B_1\) | \(6/10\) | \(0.600\) |
| \(A_2\) | \(1/10\) | \(0.100\) |
| \(B_2\) | \(0/3\) | \(0\) |
Jar \(A_1\) beats \(B_1\), and \(A_2\) beats \(B_2\). Now pour \(A_1\) and \(A_2\) into one big jar \(A\), and \(B_1\) and \(B_2\) into jar \(B\):
\[A:\ \frac{3}{13}\approx0.231,\qquad B:\ \frac{6}{13}\approx0.462.\]Combining the two better jars produces the worse jar.
Incorrect: If one doctor has the higher success rate in each type, adding the records must preserve that advantage.
Correction: Adding counts forms a weighted average of rates. The weights are each doctor’s own procedure proportions. Different weights can reverse the comparison, even though every within-type comparison points the other way.
Suppose both doctors are evaluated using the same fraction \(w\) of heart procedures and \(1-w\) of Band-Aid removals. Define standardized rates:
\[R_H(w)=w\frac79+(1-w)\cdot1,\] \[R_N(w)=w\frac15+(1-w)\frac9{10}.\]Their difference is:
\[R_H(w)-R_N(w) =w\left(\frac79-\frac15\right)+(1-w)\left(1-\frac9{10}\right) =\frac{26}{45}w+\frac1{10}(1-w)>0\]for every \(0\le w\le1\). With common weights, the higher rate in both subgroups produces a higher weighted rate.
For an equal mix, \(w=1/2\):
\[R_H(1/2)=\frac89\approx88.89\%,\qquad R_N(1/2)=\frac{11}{20}=55\%.\]These are comparisons for a specified common mix, not the observed overall success rates. Reweighting changes the question being answered.
The observed data establish subgroup rates and an aggregate reversal. They do not, by themselves, establish what would happen if the same person were assigned to either doctor. Patient selection, procedure difficulty, and other differences may matter.
Likewise, finding a reversal does not automatically mean that conditioning on every available variable gives the right comparison. The relevant conditioning depends on the question and how the data were generated.
Core idea: Identify whether the desired quantity concerns the observed mix, a common comparison mix, or a causal intervention. Those quantities need not agree.
Try these before reading the answers.
Switching loses if \(C_1\) occurs and wins if \(C_2\) or \(C_3\) occurs. The equally likely car locations give:
\[P(W)=0\frac13+1\frac13+1\frac13=\frac23.\]The host’s tie-breaking does not affect this overall result because the initial choice is wrong in exactly two of the three cases.
The likelihoods are \(P(M_3\mid C_1)=1/2\), \(P(M_3\mid C_2)=1\), and \(P(M_3\mid C_3)=0\). Hence \(P(M_3)=1/2\) and:
\[P(C_1\mid M_3)=\frac{(1/2)(1/3)}{1/2}=\frac13,\] \[P(C_2\mid M_3)=\frac{1(1/3)}{1/2}=\frac23.\]Switching to door 2 is preferable.
Here \(q=3/4\), so:
\[P(C_3\mid M_2)=\frac1{1+3/4}=\frac47.\]The probability of staying successfully after this reveal is \(3/7\). Overall, switching still succeeds whenever the first choice is wrong, so its unconditional winning probability remains \(2/3\).
The likelihood of opening door 2 and finding a goat is \(1/2\) under \(C_1\), zero under \(C_2\), and \(1/2\) under \(C_3\). The two compatible car locations have equal prior probabilities and equal likelihoods. Thus:
\[P(C_3\mid E_2)=\frac12.\]Standard Monty is forced to open door 2 when the car is behind door 3; the blind host is not. That difference changes the likelihood of the evidence.
Staying wins only when the initial choice is correct, with probability \(1/10\). Switching wins whenever that choice is wrong, with probability \(9/10\).
Heart success rates are \(70/90=7/9\) and \(2/10=1/5\). Band-Aid success rates are \(10/10=1\) and \(81/90=9/10\). Dr. Hibbert is higher in both.
Overall rates are \((70+10)/100=0.80\) and \((2+81)/100=0.83\). Heart procedures carry weights \(0.90\) for Dr. Hibbert and \(0.10\) for Dr. Nick, reversing the aggregate comparison.
For Dr. Hibbert:
\[R_H(1/4)=\frac14\frac79+\frac34 =\frac{17}{18}\approx94.44\%.\]For Dr. Nick:
\[R_N(1/4)=\frac14\frac15+\frac34\frac9{10} =\frac{29}{40}=72.5\%.\]Dr. Hibbert has the higher standardized rate.
Let subgroup rates be \(a_i>b_i\), with common weights \(w_i\ge0\) summing to 1. Then:
\[\sum_i w_i a_i-\sum_i w_i b_i =\sum_i w_i(a_i-b_i)>0.\]At least one weight is positive, and every positively weighted difference is positive. The common weighted average preserves the ordering, so a reversal cannot occur under these assumptions.
April: \(0.300>0.280\). May: \(0.200>0.100\). Player A is better in both months. Combined, A has \(23/110\approx0.209\) and B has \(29/110\approx0.264\), so B is better overall. Most of A’s at-bats came in A’s weaker month, and most of B’s in B’s stronger month.
| Concept | Essential fact |
|---|---|
| Standard Monty protocol | Host knows the car location, always reveals an unchosen goat, and always offers a switch |
| Stay | Wins exactly when the initial choice is correct: \(1/3\) |
| Switch | Wins exactly when the initial choice is wrong: \(2/3\) |
| Particular reveal | Equal tie-breaking gives \(1/3\) for the original door and \(2/3\) for the alternative |
| Unequal tie-breaking | Changes reveal-specific posteriors; overall always-switch success remains \(2/3\) |
| Blind reveal | A randomly opened door that happens to show a goat can yield \(1/2\)–\(1/2\) |
| Million doors | Extreme case that exposes the 50–50 argument; switching wins \((N-1)/N\) in general |
| Simpson reversal | Within-subgroup comparisons and aggregate comparison point in opposite directions |
| Confounder | Variable \(C\) tied to both group and outcome; here the procedure type |
| Aggregate rate | Weighted average of subgroup rates, using each group’s own case mix: \(\frac{a+c}{b+d}\), not \(\frac ab+\frac cd\) |
| Other reversals | Batting averages by half-season, Berkeley admissions by department, mixed jelly-bean jars |
| Common-weight comparison | Preserves an advantage present in every positively weighted subgroup |
| Interpretation | Observed rates, standardized rates, and causal effects answer different questions |
The rules governing which door is opened and whether a switch is offered. These rules determine the likelihoods of the observed evidence.
Choosing the sole remaining unchosen door after a standard Monty Hall goat reveal.
A reversal of a comparison after subgroup data are aggregated, despite the subgroup comparisons consistently pointing in the other direction.
A subgroup defined by a conditioning variable, such as procedure type. Comparing within strata keeps that variable fixed.
A variable associated with both the group being compared and the outcome, such as procedure type for the two doctors. Failing to condition on it can produce a misleading or reversed comparison.
The proportions of cases belonging to different subgroups. These proportions serve as the weights in an aggregate rate.
A weighted average calculated using a specified reference case mix, allowing groups to be compared under common weights.
A repeated game can be understood by conditioning on its first round: the remaining problem has the same structure, with a different starting fortune. This produces a difference equation for the probability of winning the entire game. Random variables then provide a concise way to describe numerical features of an experiment. A single success indicator is Bernoulli; a count of successes in a fixed number of independent, equally likely trials is Binomial.
| Symbol | Meaning |
|---|---|
| \(N\) | Fixed total fortune shared by two gamblers |
| \(i\) | Gambler A’s initial fortune; B starts with \(N-i\) |
| \(p\), \(q=1-p\) | Probabilities that A wins or loses one round |
| \(u_i\) | Probability A reaches \(N\) before 0, starting at \(i\) |
| \(d_i=u_i-u_{i-1}\) | Difference between neighboring winning probabilities |
| \(S\), \(s\in S\) | Sample space and an individual outcome |
| \(X:S\to\mathbb R\) | Random variable assigning a number to each outcome |
| \(X(s)\) | Value of the random variable at outcome \(s\) |
| \(\{X=x\}\) | Event consisting of outcomes mapped to \(x\) |
| \(I_A\) | Indicator of event \(A\): 1 when it occurs, 0 otherwise |
| \(X\sim\operatorname{Bern}(p)\) | Bernoulli distribution with success probability \(p\) |
| \(X\sim\operatorname{Bin}(n,p)\) | Binomial distribution: successes in \(n\) independent trials |
After studying these notes, you should be able to:
Two gamblers, A and B, repeatedly make $1 bets. A begins with \(i\) dollars and B with \(N-i\) dollars. Every round transfers one dollar between them, so the total remains \(N\). The rounds are independent, and A wins each round with the same probability \(p\); B wins with probability \(q=1-p\).
The game ends when one gambler has all the money. A’s fortune moves:
\[i\longrightarrow \begin{cases} i+1,&\text{with probability }p,\\ i-1,&\text{with probability }q. \end{cases}\]The states 0 and \(N\) are absorbing: once reached, the game is over. The question is the probability of reaching \(N\) before 0, rather than the probability of winning the next round.
Define:
\[u_i=P(\text{A reaches }N\text{ before }0\mid\text{A starts at }i).\]At the boundaries:
\[u_0=0,\qquad u_N=1.\]Core idea: The current fortune summarizes the information needed for future winning probabilities. Past wins and losses do not change future independent round probabilities.
For \(1\le i\le N-1\), the first round either raises A’s fortune to \(i+1\) or lowers it to \(i-1\). After either outcome, the remaining game has exactly the original rules with a new starting fortune.
By the law of total probability:
\[\begin{aligned} u_i &=P(\text{eventual win}\mid\text{first round won})p +P(\text{eventual win}\mid\text{first round lost})q\\ &=pu_{i+1}+qu_{i-1}. \end{aligned}\]The starting state \(i\) is understood as fixed throughout this calculation.
This is a difference equation: it relates a sequence at neighboring integer indices. The boundary conditions select the solution describing this particular game.
Incorrect: A wins the game with probability \(p\) because that is A’s chance of winning a bet.
Correction: \(p\) concerns one round. \(u_i\) concerns an entire path until absorption and depends on the starting fortune and total fortune as well as \(p\).
When \(p=q=1/2\):
\[2u_i=u_{i+1}+u_{i-1}.\]Rearrange:
\[u_{i+1}-u_i=u_i-u_{i-1}.\]Every neighboring difference is the same. Therefore \(u_i=a+bi\). The boundary \(u_0=0\) gives \(a=0\), and \(u_N=1\) gives \(b=1/N\).
A starts with $3 and B with $7, so \(N=10\). Each round is fair, but:
\[P(\text{A wins the whole game})=u_3=\frac3{10},\qquad P(\text{B wins})=\frac7{10}.\]A has fewer dollars available to absorb losses before bankruptcy. Fair individual bets do not make the two eventual winning probabilities equal when the initial fortunes differ.
Assume \(0<p<1\) and set \(r=q/p\). Because \(p+q=1\), the recurrence can be rewritten as:
\[p(u_{i+1}-u_i)=q(u_i-u_{i-1}).\]With \(d_i=u_i-u_{i-1}\), this becomes:
\[d_{i+1}=rd_i,\qquad d_i=d_1r^{i-1}.\]Starting from \(u_0=0\), add the differences:
\[u_i=d_1(1+r+\cdots+r^{i-1}).\]The other boundary condition gives:
\[1=u_N=d_1(1+r+\cdots+r^{N-1}).\]Divide these equations. For \(r\ne1\), evaluate the geometric sums:
\[u_i=\frac{1+r+\cdots+r^{i-1}}{1+r+\cdots+r^{N-1}} =\frac{1-r^i}{1-r^N}.\]For an interior starting state, \(p=0\) gives \(u_i=0\) and \(p=1\) gives \(u_i=1\) directly. The boundaries remain \(u_0=0\) and \(u_N=1\) in every case.
The geometric-sum form stays meaningful as \(r\to1\): its numerator approaches \(i\) and denominator approaches \(N\). Thus the biased formula approaches \(i/N\). The apparent \(0/0\) in the power formula at \(p=1/2\) reflects its algebraic form, not an undefined winning probability.
A standard way to solve a linear difference equation with constant coefficients is to guess a power and see what it requires. Try \(u_i=x^i\) with \(x\ne0\):
\[x^i=px^{i+1}+qx^{i-1} \quad\Longrightarrow\quad px^2-x+q=0.\]The quadratic formula gives \(x=\dfrac{1\pm\sqrt{1-4pq}}{2p}\). This simplifies nicely, because \(q=1-p\) gives
\[1-4pq=1-4p+4p^2=(2p-1)^2.\]So the two roots are
\[x=\frac{1+(2p-1)}{2p}=1,\qquad x=\frac{1-(2p-1)}{2p}=\frac qp.\]When \(p\ne q\) the roots are distinct, and the general solution is a linear combination of their powers:
\[u_i=a\cdot1^i+b\left(\frac qp\right)^i=a+b\,r^i.\]The boundary conditions fix the constants. From \(u_0=0\), \(b=-a\). From \(u_N=1\), \(a(1-r^N)=1\). Hence
\[u_i=\frac{1-r^i}{1-r^N},\]agreeing with the result above. When \(p=q\), the root 1 is repeated and the guess alone does not give a second solution. Instead, let \(x=q/p\to1\) and use L’Hôpital’s rule:
\[\lim_{x\to1}\frac{1-x^i}{1-x^N}=\lim_{x\to1}\frac{ix^{i-1}}{Nx^{N-1}}=\frac iN.\]Substituting \(u_i=i/N\) back into \(2u_i=u_{i+1}+u_{i-1}\) confirms that it satisfies the fair-game equation.
Core idea: Guessing \(x^i\) turns a difference equation into a polynomial equation. With distinct roots, the general solution is a combination of root powers, and the boundary conditions pick the particular solution.
Let A and B start with equal fortunes, \(i=N/2\), and let each round be only slightly unfavorable to A: \(p=0.49\). Then \(r=0.51/0.49\approx1.0408\), and
\[u_{N/2}=\frac{1-r^{N/2}}{1-r^N}=\frac1{1+r^{N/2}}.\]| Total fortune \(N\) | Each player starts with | \(P(\text{A wins the game})\) |
|---|---|---|
| 20 | 10 | \(\approx0.401\) |
| 100 | 50 | \(\approx0.119\) |
| 200 | 100 | \(\approx0.018\) |
A disadvantage of only 0.01 per round becomes overwhelming over a long game. With 100 dollars each, B ends up with all the money about 98% of the time.
A casino is in an even stronger position than B. Its games range from slightly to very unfavorable to the gambler, and it nearly always has far more money than the gambler. This is why the problem is called gambler’s ruin.
Let \(N=4\) and \(i=2\). If \(p=2/3\), then \(q=1/3\) and \(r=1/2\). Hence:
\[u_2=\frac{1-(1/2)^2}{1-(1/2)^4} =\frac{3/4}{15/16}=\frac45.\]A’s probability of winning one round is \(2/3\), while A’s probability of winning the whole game is \(4/5\). If instead \(p=1/3\), then \(r=2\) and:
\[u_2=\frac{1-2^2}{1-2^4}=\frac{-3}{-15}=\frac15.\]The negative numerator and denominator cancel; the probability is positive.
Nothing in the rules prevents the game from going on forever. The probability of that is zero, and the formulas already show it.
The probability that B wins comes from the same calculation with the roles exchanged. B wins each round with probability \(q\) and starts with \(N-i\), so swap \(p\) with \(q\) and \(i\) with \(N-i\).
Fair game. B wins with probability \((N-i)/N\), so
\[P(\text{A wins})+P(\text{B wins})=\frac iN+\frac{N-i}N=1.\]Biased game. With \(r=q/p\), B’s ratio is \(p/q=1/r\):
\[P(\text{B wins})=\frac{1-r^{-(N-i)}}{1-r^{-N}} =\frac{r^N-r^i}{r^N-1} =\frac{r^i-r^N}{1-r^N},\]after multiplying numerator and denominator by \(r^N\). Adding A’s probability:
\[\frac{1-r^i}{1-r^N}+\frac{r^i-r^N}{1-r^N}=1.\]In both cases the two ways for the game to end already have total probability 1, leaving no probability for endless play.
For \(0<p<1\), from any interior state, \(N\) consecutive wins would force absorption at \(N\) within those rounds. The probability of that run of potential round outcomes is \(p^N>0\). Thus, conditional on any history that has not yet ended, the probability of surviving the next \(N\) rounds is at most \(1-p^N\).
After \(m\) such blocks:
\[P(\text{not absorbed after }mN\text{ rounds})\le(1-p^N)^m\longrightarrow0.\]At \(p=0\) or \(p=1\), absorption occurs deterministically within \(N\) rounds. This argument establishes eventual absorption without solving for the winning probabilities; it does not calculate the expected duration.
A random variable is a function assigning a real number to every outcome in the sample space:
\[X:S\longrightarrow\mathbb R.\]The experiment selects a random outcome \(s\) according to the probability model. The rule \(X\) is fixed; its realized value is \(X(s)\). An outcome need not be a number: it may be a sequence of coin tosses, a path of fortunes, or another experimental record.
Assume independent tosses. The four outcomes \(HH,HT,TH,TT\) each have probability \(1/4\). Define \(X\) as the number of Heads, \(Y\) as the number of Tails, and \(I\) as the indicator that the first toss is Heads.
| Outcome \(s\) | \(X(s)\) | \(Y(s)\) | \(I(s)\) |
|---|---|---|---|
| \(HH\) | 2 | 0 | 1 |
| \(HT\) | 1 | 1 | 1 |
| \(TH\) | 1 | 1 | 0 |
| \(TT\) | 0 | 2 | 0 |
The relationship \(Y=2-X\) holds outcome by outcome. The statement \(X=1\) describes the event \(\{HT,TH\}\), so:
\[P(X=1)=P(\{HT,TH\})=\frac12.\]In general:
\[\{X=x\}=\{s\in S:X(s)=x\}.\]Core idea: A numerical value can summarize several distinct outcomes. Its probability is the combined probability of all outcomes mapped to it.
Incorrect: The three possible Head counts, 0, 1, and 2, are equally likely.
Correction: They collect different numbers of equally likely outcomes. Their probabilities are \(1/4,1/2,1/4\), respectively. Equal likelihood of sample outcomes does not automatically transfer to numerical summaries.
A distribution specifies the probabilities associated with a random variable’s possible values. In this example, \(X\) and \(Y\) have the same distribution: each has probabilities \(1/4,1/2,1/4\) at 0, 1, and 2. They are different functions, since \(X(HH)=2\) but \(Y(HH)=0\).
Knowing a distribution does not identify the outcome-by-outcome function or the dependence between two random variables. Here \(X+Y=2\) always, so the counts are dependent despite having identical distributions.
Let \(X_t\) be A’s fortune after \(t\) rounds, keeping the fortune fixed after absorption. Then B’s fortune is \(N-X_t\), and the fortune difference is \(2X_t-N\). These are numerical functions of the same game path. Random-variable notation expresses their relationship directly.
For an event \(A\), its indicator is:
\[I_A(s)= \begin{cases} 1,&s\in A,\\ 0,&s\notin A. \end{cases}\]If \(P(A)=p\), then:
\[P(I_A=1)=p,\qquad P(I_A=0)=1-p.\]We write \(I_A\sim\operatorname{Bern}(p)\). The symbol \(\sim\) means “is distributed as”; it does not mean equality of functions.
“Success” is simply the outcome labeled 1. It could represent Heads, a correct answer, or the occurrence of a defect. A Bernoulli model describes one binary indicator even if the underlying experiment has many possible outcomes.
For example, let \(J\) indicate that A wins the entire gambling game starting from \(i\). Then \(J\sim\operatorname{Bern}(u_i)\). The indicator of winning one round has parameter \(p\) instead. Both are Bernoulli, but they summarize different events.
The usual nondegenerate case has \(0<p<1\). Allowing \(p=0\) or \(p=1\) gives a constant random variable.
Suppose there are:
If \(I_j\) indicates success on trial \(j\), then:
\[X=I_1+\cdots+I_n\]counts successes, and \(X\sim\operatorname{Bin}(n,p)\). One trial is the special case \(\operatorname{Bin}(1,p)=\operatorname{Bern}(p)\) in distribution.
One specified sequence containing \(k\) successes and \(n-k\) failures has probability:
\[p^k(1-p)^{n-k}.\]There are \(\binom nk\) sequences of this kind: choose the \(k\) positions occupied by successes. The sequences are disjoint, so their probabilities add:
The probability is zero outside these integer values. This function is called the probability mass function (PMF). For \(0<p<1\), all integers from 0 through \(n\) are possible with positive probability. At the endpoints, \(p=0\) concentrates at 0 and \(p=1\) concentrates at \(n\).
The binomial theorem verifies that the probabilities sum to 1:
\[\sum_{k=0}^n\binom nk p^k(1-p)^{n-k} =(p+(1-p))^n=1.\]This is the probability formula needed for the Binomial story; a broader treatment of PMFs and CDFs follows in the next lecture.
If \(X\sim\operatorname{Bin}(n,p)\) and \(Y\sim\operatorname{Bin}(m,p)\) are independent, then
\[\boxed{X+Y\sim\operatorname{Bin}(n+m,p).}\]Story proof. Let \(X\) count successes in \(n\) independent Bernoulli(\(p\)) trials, and let \(Y\) count successes in \(m\) further trials, independent of the first \(n\). Together these are \(n+m\) independent trials, each with success probability \(p\). The total number of successes is \(X+Y\), which by definition is \(\operatorname{Bin}(n+m,p)\). No equations are needed.
The same \(p\) is essential. If \(X\sim\operatorname{Bin}(n,1/2)\) and \(Y\sim\operatorname{Bin}(m,1/3)\), the combined trials do not share one success probability, and \(X+Y\) is not Binomial in general. Algebraic proofs follow in the next lecture.
Suppose four attempts are independent and each succeeds with probability \(1/3\). Let \(X\) count successes. Then \(X\sim\operatorname{Bin}(4,1/3)\), and:
\[P(X=2)=\binom42\left(\frac13\right)^2\left(\frac23\right)^2 =6\frac19\frac49=\frac8{27}.\]The sequence success, success, failure, failure is only one of six arrangements with two successes. Its probability is \(4/81\), while the event \(X=2\) combines all six.
For at least one success, use the complement:
\[P(X\ge1)=1-P(X=0)=1-\left(\frac23\right)^4=\frac{65}{81}.\]Incorrect: Any count of successes in \(n\) trials is Binomial.
Correction: Independence and a common success probability are part of the model. If trials have different success probabilities, or one trial’s outcome changes the others, the Binomial formula is not automatically valid.
Toss one fair coin and copy its result into two trial records. Each record individually has success probability \(1/2\), but the records are perfectly dependent. The total success count is 0 or 2, each with probability \(1/2\), and never 1.
A \(\operatorname{Bin}(2,1/2)\) count would instead assign probability \(1/2\) to 1. Identical Bernoulli marginal distributions do not establish independent trials.
If independent potential gambling rounds are generated for a fixed \(n\), their win count is \(\operatorname{Bin}(n,p)\). The actual gambler’s ruin game stops at a random time determined by its outcomes. The number of wins before bankruptcy or total victory is therefore not justified as a fixed-\(n\) Binomial count.
Core idea: Identify what is counted and when observation stops. The distribution story depends on the experiment’s rules.
Try these before reading the answers.
Conditioning on the first-round outcome gives \(u_i=pu_{i+1}+qu_{i-1}\) for interior states. Independence and the unchanged round probability make the remaining game equivalent to a fresh game at the new fortune. A has already lost at 0 and already won at \(N\), giving \(u_0=0\) and \(u_N=1\).
Here \(N=10\). A wins with probability \(u_4=4/10=2/5\). Since the finite game ends with probability 1, B wins with probability \(1-2/5=3/5\).
With \(r=q/p=1/2\):
\[u_1=\frac{1-1/2}{1-(1/2)^3}=\frac{1/2}{7/8}=\frac47.\]This is less than \(2/3\). A has a favorable per-round probability but only one dollar of starting fortune; eventual victory requires reaching $3 before losing that dollar.
Here \(r=2\) and \(1-r^4=-15\). Therefore:
\[u_1=\frac1{15},\qquad u_2=\frac3{15}=\frac15,\qquad u_3=\frac7{15}.\]At \(i=2\):
\[pu_3+qu_1=\frac13\frac7{15}+\frac23\frac1{15} =\frac9{45}=\frac15=u_2.\]Exactly one Head with the first toss Heads occurs only for \(HT\). Hence \(\{X=1,I=1\}=\{HT\}\) and its probability is \(1/4\).
Let \(J=1\) for \(HH,HT,TH\) and 0 for \(TT\). Then \(P(J=1)=3/4\), so \(J\sim\operatorname{Bern}(3/4)\). A two-toss experiment can still yield a Bernoulli random variable when the summary is binary.
The factor 10 counts the choices of two success positions among five trials. Also:
\[P(X\ge1)=1-P(X=0)=1-\frac1{32}=\frac{31}{32}.\]The records are dependent. Their count has probabilities \(P(X=0)=1/2\), \(P(X=2)=1/2\), and \(P(X=1)=0\), which differs from the Binomial distribution.
The number of trials is determined by the game’s outcomes, rather than fixed in advance. Independence of potential round outcomes alone does not supply the fixed-trial Binomial story for a count taken up to that stopping time.
The guess gives \(\frac23x^2-x+\frac13=0\), or \(2x^2-3x+1=0\), with roots \(x=1\) and \(x=\frac12=q/p\). So \(u_i=a+b(1/2)^i\). Then \(u_0=0\) gives \(b=-a\), and \(u_3=1\) gives \(a(1-1/8)=1\), so \(a=8/7\) and
\[u_i=\frac87\left(1-2^{-i}\right),\qquad u_1=\frac47.\]B wins each round with probability \(1/3\) and starts with 2, so B’s ratio is \(p/q=2\) and B wins with probability \(\dfrac{1-2^2}{1-2^3}=\dfrac37\). Indeed \(\frac47+\frac37=1\).
| Concept | Essential fact |
|---|---|
| Gambler’s ruin | Fortune moves by one dollar until reaching 0 or \(N\) |
| One round versus whole game | \(p\) and \(u_i\) describe different events |
| First-step recurrence | \(u_i=pu_{i+1}+qu_{i-1}\) |
| Boundaries | \(u_0=0\) and \(u_N=1\) |
| Fair game | \(u_i=i/N\) |
| Biased game | \(u_i=[1-(q/p)^i]/[1-(q/p)^N]\) for \(0<p<1\), \(p\ne1/2\) |
| Neighboring differences | \(d_{i+1}=(q/p)d_i\) |
| Characteristic roots | Guess \(u_i=x^i\): \(px^2-x+q=0\) has roots 1 and \(q/p\), so \(u_i=a+b(q/p)^i\) |
| Slightly unfair game | \(p=0.49\), equal fortunes: A wins \(\approx0.40,0.12,0.02\) for \(N=20,100,200\) |
| Finite-game termination | \(P(\text{A wins})+P(\text{B wins})=1\), so the game ends with probability 1 |
| Random variable | A fixed function \(X:S\to\mathbb R\) |
| Numerical event | \(\{X=x\}=\{s:X(s)=x\}\) |
| Same distribution | Same value probabilities; functions may differ |
| Bernoulli | Indicator with probabilities \(p\) at 1 and \(1-p\) at 0 |
| Binomial story | Fixed \(n\), independent binary trials, common \(p\) |
| Binomial formula | \(P(X=k)=\binom nk p^k(1-p)^{n-k}\) |
| Sum of Binomials | Independent \(\operatorname{Bin}(n,p)+\operatorname{Bin}(m,p)\sim\operatorname{Bin}(n+m,p)\) by the story |
A state at which the process stops or remains permanently. In gambler’s ruin, these are fortunes 0 and \(N\).
Conditioning on the first transition to express a probability in terms of the same problem at new starting states.
An equation relating a sequence at different integer indices. Boundary conditions determine the desired solution here.
A real-valued function on the sample space. Its realized value depends on the outcome selected by the experiment.
The probabilities of the numerical events associated with a random variable.
A random variable equal to 1 when a specified event occurs and 0 otherwise.
The distribution of a success indicator, with probability \(p\) at 1 and \(1-p\) at 0.
The distribution of the number of successes in a fixed number \(n\) of independent Bernoulli trials sharing success probability \(p\).
For a discrete random variable, the function assigning \(P(X=x)\) to each possible numerical value \(x\).
A random variable summarizes an outcome numerically; its distribution describes the probabilities of those numerical values. For a discrete random variable, the PMF gives probability at each value and the CDF accumulates probability up to a threshold. Their connection makes the jumps of a CDF meaningful. Sampling without replacement gives a different success-count story from independent trials: the Hypergeometric distribution accounts for the finite supply of successes and failures.
| Symbol | Meaning |
|---|---|
| \(X\), \(x\) | Random variable and a numerical value or threshold |
| \(\{X=x\}\) | Event of outcomes mapped to \(x\) |
| \(p_X(x)\) | PMF: \(P(X=x)\) for a discrete random variable |
| \(F_X(x)\) | CDF: \(P(X\le x)\) for any random variable |
| \(F_X(a^-)\) | Left limit at \(a\): \(P(X<a)\) |
| \(\operatorname{supp}(X)\) | Discrete support: values with positive probability |
| \(X\sim\operatorname{Bin}(n,p)\) | Success count in \(n\) independent trials with common probability \(p\) |
| \(w,b\) | Numbers of success-type and failure-type items in a population |
| \(n\) | Number of items sampled without replacement |
| \(X\sim\operatorname{HGeom}(w,b,n)\) | Number of success-type items in a uniform sample of size \(n\) |
| \(\binom ak\) | Number of subsets of size \(k\) from \(a\) distinct items |
After studying these notes, you should be able to:
A random variable is a fixed numerical function of the outcome of an experiment. Its distribution describes the probabilities of events involving its values. Several distinct outcomes may produce the same value, and two different random variables may have the same distribution.
For two independent fair coin tosses, let \(X\) count Heads and \(Y\) count Tails. Then \(Y=2-X\), and both variables have probabilities \(1/4,1/2,1/4\) at 0, 1, and 2. Yet \(X(HH)=2\) and \(Y(HH)=0\): they are different functions.
The notation \(P(X=x)\) refers to an event:
\[\{X=x\}=\{s\in S:X(s)=x\}.\]Writing \(P(X)\) alone does not identify an event whose probability is being requested.
Core idea: The distribution retains the probabilities of numerical summaries. It need not retain the full experimental record or identify the function that created the summary.
A random variable is discrete if its probability is concentrated on a finite or countably infinite set of values. These values need not be integers. For example, a variable taking values \(-2,1/2,\sqrt2\) is discrete.
For a discrete random variable:
\[p_X(x)=P(X=x).\]Its support is the set of values at which \(p_X(x)>0\). A valid PMF must be nonnegative and sum to 1 over the distinct support values:
\[p_X(x)\ge0,\qquad \sum_{x\in\operatorname{supp}(X)}p_X(x)=1.\]The events \(\{X=x\}\) are disjoint and exhaust the possible values with probability 1. Thus for a set \(A\) of numerical values:
\[P(X\in A)=\sum_{x\in A\cap\operatorname{supp}(X)}p_X(x).\]A PMF plot shows probability through the heights of the mass markers or vertical lines. Summing those heights gives total probability; ordinary area beneath the connecting lines is not the relevant quantity.
Roll two independent fair six-sided dice and let \(T\) be their sum. The 36 ordered outcomes are equally likely. A total of 2 comes only from \((1,1)\); a total of 7 comes from six outcomes, \((1,6),(2,5),\ldots,(6,1)\).
| \(t\) | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| \(36p_T(t)\) | 1 | 2 | 3 | 4 | 5 | 6 | 5 | 4 | 3 | 2 | 1 |
The support is \(\{2,3,\ldots,12\}\). The numerators sum to 36, verifying normalization. For example:
\[P(T\le4)=\frac{1+2+3}{36}=\frac16.\]Core idea: Equally likely ordered dice outcomes produce unequal probabilities for the sum because different totals collect different numbers of outcomes.
Incorrect: The sum has eleven possible values, so each has probability \(1/11\).
Correction: Counting possible values does not establish equal probability. Count or otherwise measure the underlying outcomes mapped to each value.
The cumulative distribution function is:
The CDF is defined at every real threshold \(x\), including thresholds that the random variable cannot attain. For a discrete variable:
\[F_X(x)=\sum_{t\le x}p_X(t),\]where only support values contribute. A threshold of 1.5 can therefore have a nonzero CDF value even when \(P(X=1.5)=0\).
Let \(X\sim\operatorname{Bin}(4,1/2)\). The PMF is:
\[p_X(k)=\frac{\binom4k}{16},\qquad k=0,1,2,3,4.\]| \(k\) | 0 | 1 | 2 | 3 | 4 |
|---|---|---|---|---|---|
| \(p_X(k)\) | \(1/16\) | \(4/16\) | \(6/16\) | \(4/16\) | \(1/16\) |
| \(F_X(k)\) | \(1/16\) | \(5/16\) | \(11/16\) | \(15/16\) | \(1\) |
The full CDF, including noninteger thresholds, is:
\[F_X(x)= \begin{cases} 0,&x<0,\\ 1/16,&0\le x<1,\\ 5/16,&1\le x<2,\\ 11/16,&2\le x<3,\\ 15/16,&3\le x<4,\\ 1,&x\ge4. \end{cases}\]Hence \(F_X(1.5)=5/16\) because the allowed counts up to 1.5 are 0 and 1. At \(x=2\), the CDF includes the probability at 2 and rises to \(11/16\).
Core idea: A PMF assigns probability separately; a CDF accumulates it. They describe the same discrete distribution in different forms.
Every CDF has three defining properties:
A CDF can have flat regions and jumps; it need not increase strictly or be continuous everywhere.
Define the left limit \(F_X(a^-)=\lim_{x\uparrow a}F_X(x)=P(X<a)\). Then:
\[P(X=a)=F_X(a)-F_X(a^-).\]For an integer-valued variable and integer \(k\), this simplifies to:
\[P(X=k)=F_X(k)-F_X(k-1).\]The integer formula should not be applied blindly to a variable with noninteger support.
For \(a<b\):
| Requested probability | CDF expression |
|---|---|
| \(P(a<X\le b)\) | \(F_X(b)-F_X(a)\) |
| \(P(a\le X\le b)\) | \(F_X(b)-F_X(a^-)\) |
| \(P(X>a)\) | \(1-F_X(a)\) |
| \(P(X\ge a)\) | \(1-F_X(a^-)\) |
For the four-toss example:
\[P(1<X\le3)=\frac{15}{16}-\frac5{16}=\frac58,\]whereas:
\[P(1\le X\le3)=\frac{15}{16}-\frac1{16}=\frac78.\]The difference is exactly \(P(X=1)=4/16\).
Incorrect: \(P(X\ge a)=1-F_X(a)\) for every random variable.
Correction: That expression gives \(P(X>a)\). If there is positive probability at \(a\), including or excluding the endpoint changes the answer.
A discrete variable has probability mass at individual values. A continuous distribution has a continuous CDF and no positive probability at a single point. Its probability is assigned to ranges of values.
For example, a continuously uniform location on \([0,1]\) has:
\[F_X(x)= \begin{cases} 0,&x<0,\\ x,&0\le x\le1,\\ 1,&x>1. \end{cases}\]Then \(P(1/4<X\le3/4)=1/2\), but \(P(X=1/2)=0\). Observing a particular value does not require that value to have positive point probability.
The CDF works for both discrete and continuous distributions. A PMF describes discrete probability mass; continuous distributions with densities are treated in more detail later. Mixed distributions can contain both point masses and a continuous component.
Core idea: Zero probability at each individual point is compatible with positive probability over an interval containing infinitely many points.
For \(n\) independent Bernoulli trials with common success probability \(p\):
\[X=I_1+\cdots+I_n\sim\operatorname{Bin}(n,p),\qquad P(X=k)=\binom nk p^k(1-p)^{n-k}.\]The story, indicator sum, and PMF are three descriptions of the same model.
The indicators \(I_1,\ldots,I_n\) are i.i.d., short for independent and identically distributed:
Identically distributed does not mean equal. \(I_1\) indicates success on the first trial and \(I_2\) on the second; they are different random variables, and in a given experiment one may be 1 while the other is 0. They share only their distribution: the same probabilities of taking each value. Many different random variables can have the same distribution, just as many houses can be built from one blueprint.
If \(X\sim\operatorname{Bin}(n,p)\) and \(Y\sim\operatorname{Bin}(m,p)\) are independent, then:
\[\boxed{X+Y\sim\operatorname{Bin}(n+m,p).}\]Here \(X+Y\) is a sum of two functions on the same sample space: for each outcome, compute both and add.
1. Story. Let \(X\) count successes in \(n\) independent trials and \(Y\) count successes in \(m\) further independent trials, all with success probability \(p\). Then \(X+Y\) counts the successes in \(n+m\) such trials, so it is \(\operatorname{Bin}(n+m,p)\).
2. Indicator sums. Write \(X=X_1+\cdots+X_n\) and \(Y=Y_1+\cdots+Y_m\), where all the \(X_j\) and \(Y_j\) are i.i.d. \(\operatorname{Bern}(p)\). Then \(X+Y\) is a sum of \(n+m\) i.i.d. \(\operatorname{Bern}(p)\) indicators, which is \(\operatorname{Bin}(n+m,p)\).
3. PMF. Condition on \(X=j\) by LOTP. Given \(X=j\), the event \(X+Y=k\) is the event \(Y=k-j\), and by independence conditioning on \(X\) does not change the distribution of \(Y\). With impossible terms interpreted as zero:
\[\begin{aligned} P(X+Y=k) &=\sum_jP(X=j)P(Y=k-j)\\ &=p^k(1-p)^{n+m-k}\sum_j\binom nj\binom m{k-j}\\ &=\binom{n+m}{k}p^k(1-p)^{n+m-k}. \end{aligned}\]The last step uses Vandermonde’s identity. A sum of this kind, the distribution of a sum of independent random variables, is called a convolution. This route takes the most work, and it would get stuck without Vandermonde. Read the other way, since the story proof already guarantees the answer, the calculation is a second proof of Vandermonde’s identity.
Independence and a common \(p\) are essential for this conclusion; two Binomial marginal distributions alone do not establish it.
Core idea: Thinking of a Binomial as a story or as a sum of indicators often replaces a hard calculation with one line.
A population contains \(w\) success-type items and \(b\) failure-type items. Choose a sample of size \(n\) uniformly without replacement, meaning every subset of size \(n\) is equally likely. Let \(X\) count success-type items in the sample:
\[X\sim\operatorname{HGeom}(w,b,n).\]Assume \(w,b,n\) are nonnegative integers, \(w+b>0\), and \(0\le n\le w+b\).
There are \(\binom{w+b}{n}\) possible samples. A sample with exactly \(k\) successes must contain \(k\) of the \(w\) success items and \(n-k\) of the \(b\) failure items. Therefore:
The feasible integer values satisfy:
\[0\le k\le w,\qquad 0\le n-k\le b,\]or equivalently:
\[\boxed{\max(0,n-b)\le k\le\min(n,w).}\]The PMF is zero outside this range. If the sample is larger than the supply of failures, some successes are unavoidable. If the success supply is smaller than the sample, the count cannot exceed that supply.
Vandermonde’s identity verifies normalization:
\[\sum_k\binom wk\binom b{n-k}=\binom{w+b}n.\]Core idea: Count the successful sample subsets, then divide by all equally likely sample subsets. The finite supplies determine both the numerator and the feasible range.
An urn contains four white balls and two black balls. Select three uniformly without replacement, and let \(X\) count white balls. Then \(X\sim\operatorname{HGeom}(4,2,3)\).
Since only two black balls exist, at least one white ball must be selected. The support is \(\{1,2,3\}\). There are \(\binom63=20\) samples:
| \(k\) | Successful subsets | \(P(X=k)\) |
|---|---|---|
| 1 | \(\binom41\binom22=4\) | \(1/5\) |
| 2 | \(\binom42\binom21=12\) | \(3/5\) |
| 3 | \(\binom43\binom20=4\) | \(1/5\) |
Thus \(P(X\ge2)=3/5+1/5=4/5\). Its CDF is:
\[F_X(x)= \begin{cases} 0,&x<1,\\ 1/5,&1\le x<2,\\ 4/5,&2\le x<3,\\ 1,&x\ge3. \end{cases}\]The CDF jumps recover the three PMF values, connecting the sampling story to both distribution representations.
With replacement, each draw from the urn has success probability \(w/(w+b)\) and, under independent random draws, the success count is:
\[\operatorname{Bin}\left(n,\frac{w}{w+b}\right).\]Without replacement, successive outcomes are dependent. In the four-white, two-black urn:
\[P(\text{second white}\mid\text{first white})=\frac35,\]but:
\[P(\text{second white}\mid\text{first black})=\frac45.\]The second draw’s unconditional white probability is still \(4/6\) by symmetry. Equal marginal success probabilities do not imply independence.
| Feature | Binomial | Hypergeometric |
|---|---|---|
| Basic experiment | Independent binary trials | Uniform sample from a finite population |
| Urn interpretation | With replacement | Without replacement |
| Dependence | Trials independent | Draws generally dependent |
| Parameters | \(n,p\) | \(w,b,n\) |
| Support | \(0,\ldots,n\) for \(0<p<1\) | \(\max(0,n-b),\ldots,\min(n,w)\) |
Incorrect: The initial white-ball proportion is \(2/3\), so three draws without replacement give \(\operatorname{Bin}(3,2/3)\).
Correction: The composition changes after each draw. The Hypergeometric model gives \(P(X=3)=1/5\), while the with-replacement Binomial model gives \((2/3)^3=8/27\). They describe different experiments.
If the population is huge compared with the sample, for example 10 marbles drawn from a billion, it is extremely unlikely that the same item would be picked twice even with replacement. Sampling with and without replacement then behave almost identically, so
\[\operatorname{HGeom}(w,b,n)\approx\operatorname{Bin}\left(n,\frac{w}{w+b}\right) \quad\text{when }w+b\text{ is much larger than }n.\]For example, keep the white fraction at \(0.6\) and draw \(n=5\). The probability that all five are white is:
| Population size \(w+b\) | Hypergeometric | \(\operatorname{Bin}(5,0.6)\) |
|---|---|---|
| 10 | \(0.0238\) | \(0.0778\) |
| 100 | \(0.0725\) | \(0.0778\) |
| 1000 | \(0.0772\) | \(0.0778\) |
| 1,000,000 | \(0.07776\) | \(0.07776\) |
In a small population, each white draw noticeably depletes the white balls. In a large one, it hardly matters.
The number of aces in a hand is a natural count that is not Binomial. Think of each card as a trial: the trials are not independent, because each ace dealt makes later aces less likely, and if the first four cards are aces the fifth cannot be. The possible values are 0, 1, 2, 3, and 4.
A uniformly chosen five-card hand from a standard 52-card deck contains four possible aces and 48 non-aces. If \(X\) counts aces, then:
\[X\sim\operatorname{HGeom}(4,48,5),\qquad P(X=3)=\frac{\binom43\binom{48}2}{\binom{52}5} =\frac{4512}{2598960}\approx0.001736.\]This is approximately \(0.1736\%\). Selecting which three aces and which two non-aces gives the favorable hands; ordering within a hand is irrelevant.
A population has \(N\) animals, of which \(m\) have been tagged. A later sample of \(n\) distinct animals is uniformly selected. If \(X\) counts tagged animals in the sample:
\[X\sim\operatorname{HGeom}(m,N-m,n).\]For \(N=20\), \(m=5\), and \(n=4\):
\[P(X=2)=\frac{\binom52\binom{15}2}{\binom{20}4} =\frac{1050}{4845}=\frac{70}{323}\approx0.2167.\]Uniform sampling assumes that tagged and untagged animals are sampled under the same mechanism; tagging must not alter recapture chances for this model.
Core idea: White balls, aces, and tagged animals share the same structure: a fixed success subset intersected with a uniform sample subset.
Try these before reading the answers.
The total mass is \(4c=1\), so \(c=1/4\). The support is \(\{-1,0,2\}\) and \(P(X>0)=P(X=2)=1/4\).
Thus \(F_X(1)=3/4\) and the jump at 2 is \(1-3/4=1/4=P(X=2)\).
The first two differ by the mass \(P(X=2)=6/16\). For the last probability, subtract \(P(X<2)\), leaving the endpoint mass included.
No. A CDF is nondecreasing, but the candidate drops from 0.6 to 0.4 as the threshold increases from 0 to 1. The events cannot lose probability as more values become included.
Probability is countably additive, not additive through an unrestricted sum over uncountably many individual points. The interval probability is determined by the CDF: \(F_X(1)-F_X(0)=1\). The endpoints have zero mass, so strict inequalities give the same result here.
All integers in that range are feasible, and:
\[P(X=2)=\frac{\binom32\binom72}{\binom{10}4} =\frac{63}{210}=\frac3{10}.\]Five sampled items can include at most two black balls, so at least three must be white. The support is \(\{3,4,5\}\) and a white count of 2 is impossible.
\(F_X(2.5)=P(X\le2)=1/5+3/5=4/5\). Also \(P(X>1)=1-F_X(1)=4/5\). These happen to match because the support is \(\{1,2,3\}\).
The sum is \(\operatorname{Bin}(5,1/4)\). Zero successes means failure in all five trials:
\[P(X+Y=0)=\left(\frac34\right)^5=\frac{243}{1024}.\]| Concept | Essential fact |
|---|---|
| Random variable | Numerical function of an outcome |
| Distribution | Probabilities of numerical events |
| Discrete support | Values with \(P(X=x)>0\) |
| PMF | \(p_X(x)=P(X=x)\); nonnegative and sums to 1 |
| CDF | \(F_X(x)=P(X\le x)\), defined for every real threshold |
| Valid CDF | Nondecreasing, right-continuous, limits 0 and 1 |
| PMF from CDF | \(P(X=a)=F_X(a)-F_X(a^-)\) |
| Integer-valued case | \(P(X=k)=F_X(k)-F_X(k-1)\) for integer \(k\) |
| Interval | \(P(a<X\le b)=F_X(b)-F_X(a)\) |
| Strict upper tail | \(P(X>a)=1-F_X(a)\) |
| Continuous point probability | No mass at a single point; ranges can have positive probability |
| Hypergeometric story | Uniform sample of \(n\) items without replacement |
| Hypergeometric PMF | \(\binom wk\binom b{n-k}/\binom{w+b}n\) |
| Hypergeometric support | \(\max(0,n-b)\le k\le\min(n,w)\), integer \(k\) |
| Binomial comparison | Independent trials versus dependent finite-population draws |
| Large population | \(\operatorname{HGeom}(w,b,n)\approx\operatorname{Bin}(n,w/(w+b))\) when \(w+b\gg n\) |
| i.i.d. | Independent and identically distributed; same distribution, not the same random variable |
| Sum of Binomials | Independent \(\operatorname{Bin}(n,p)+\operatorname{Bin}(m,p)\sim\operatorname{Bin}(n+m,p)\): story, indicators, or convolution with Vandermonde |
For a discrete random variable, the set of numerical values with positive probability.
The function giving the probability of each individual value of a discrete random variable.
The function giving the probability of falling at or below each real threshold. It describes discrete, continuous, and mixed distributions.
The difference between the CDF value at a point and its left limit, equal to the probability mass at that point.
The CDF value at a point equals the limit approached from larger thresholds; a mass point is included at its own threshold.
The distribution of a success count in a uniform sample of fixed size drawn without replacement from a finite population with known success and failure counts.
Selecting distinct population items so a selected item cannot be drawn again. The remaining composition depends on earlier draws.
Independent and identically distributed: random variables that are independent and all have the same distribution, such as the success indicators of Binomial trials.
The distribution of a sum of independent random variables, computed by conditioning on one of them and summing (or integrating) over its values.
Expectation compresses a distribution into a probability-weighted average. Its most useful property is linearity: expectations of sums add even when the summands are dependent. Indicators turn counts into sums of simple yes/no variables, and their expectations are probabilities. Together with symmetry, this connects probability calculations to expected counts without requiring the full distribution of the count. Geometric waiting times provide an example where the counting convention matters by exactly one trial.
| Symbol | Meaning |
|---|---|
| \(X\perp Y\) | Independent random variables |
| \(p\), \(q=1-p\) | Success and failure probabilities in repeated independent trials |
| \(G\sim\operatorname{Geom}(p)\) | Number of failures before the first success |
| \(T=G+1\) | Total trials through and including the first success |
| \(E(X)\) | Expected value, or mean, of \(X\) |
| \(\sum_x xP(X=x)\) | Probability-weighted average over distinct support values |
| \(I_A\) | Indicator of event \(A\) |
| \(E(I_A)=P(A)\) | Fundamental bridge between expectation and probability |
| \(C=\sum_j I_j\) | A count written as a sum of indicators |
| \(w,b,n\) | Success population, failure population, and sample size in a Hypergeometric model |
After studying these notes, you should be able to:
Random variables \(X\) and \(Y\) are independent when learning the value of one does not change the distribution of the other. A definition that works generally is:
\[P(X\le x,Y\le y)=P(X\le x)P(Y\le y)\]for all real \(x,y\). For discrete variables, an equivalent condition is:
\[P(X=x,Y=y)=P(X=x)P(Y=y)\]for every pair of support values. Where \(P(X=x)>0\), this gives \(P(Y=y\mid X=x)=P(Y=y)\).
Having the same distribution is a different property. If one fair coin is tossed and \(X\) indicates Heads while \(Y=1-X\) indicates Tails, both are \(\operatorname{Bern}(1/2)\). But:
\[P(X=1,Y=1)=0\ne\frac12\frac12.\]They are dependent. Indicators from two independent fair tosses also have the same distribution, but are independent.
For several variables, mutual independence requires the analogous factorization for all value combinations. Pairwise independence alone is weaker. The abbreviation i.i.d. means independent and identically distributed: it specifies both properties.
Core idea: Marginal distributions tell how each variable behaves separately. Independence specifies how their outcomes occur together.
Perform independent Bernoulli trials, each with success probability \(p\), where \(0<p<1\). Stop at the first success. Let \(G\) count the failures before that success:
\[G\sim\operatorname{Geom}(p),\qquad G\in\{0,1,2,\ldots\}.\]The event \(G=k\) requires exactly \(k\) failures followed by one success. Independence gives:
There is no binomial coefficient: stopping at the first success fixes the order. Summing the geometric series verifies normalization:
\[\sum_{k=0}^{\infty}q^kp=\frac{p}{1-q}=1.\]The probability of no success in the first \(m\) trials is \(q^m\to0\). Thus a success eventually occurs with probability 1, although there is no finite bound on the possible waiting time.
Let \(T=G+1\) count all trials, including the success. The textbook calls this the First Success distribution. Then:
\[P(T=t)=q^{t-1}p,\qquad t=1,2,3,\ldots.\]| Quantity counted | Smallest value | Probability formula |
|---|---|---|
| Failures before success, \(G\) | 0 | \(P(G=k)=q^kp\) |
| Trials including success, \(T\) | 1 | \(P(T=t)=q^{t-1}p\) |
Roll a fair die independently until the first six. Here \(p=1/6\) and \(q=5/6\). A first six on roll 4 means three non-sixes followed by a six:
\[P(T=4)=P(G=3)=\left(\frac56\right)^3\frac16=\frac{125}{1296}.\]At least four rolls are required precisely when the first three rolls are non-sixes:
\[P(T\ge4)=\left(\frac56\right)^3=\frac{125}{216}.\]For a discrete random variable with distinct support values:
| Each possible value receives weight equal to its probability. For the finite expectations used here, the sum is well-defined; with an infinite support, absolute convergence, $$\sum_x | x | P(X=x)<\infty$$, guarantees a finite expectation. A nonnegative variable can also have an infinite mean. |
For a fair die result \(D\):
\[E(D)=\frac{1+2+3+4+5+6}{6}=\frac72=3.5.\]No die roll equals 3.5. The mean describes the center of the distribution, not a guaranteed or necessarily possible observation.
Suppose \(X\) takes values \(-2,1,4\) with probabilities \(1/4,1/2,1/4\). Then:
\[E(X)=(-2)\frac14+1\frac12+4\frac14=1.\]The simple average of the three distinct values happens also to be 1 here, but that coincidence does not justify ignoring the weights. For example, changing the probabilities to \(1/2,1/4,1/4\) changes the expectation to \(1/4\) while leaving the possible values unchanged.
The expectation depends only on the distribution. Equal distributions imply equal expectations when finite. The converse fails: a constant 1 and a variable taking 0 or 2 with equal probability both have mean 1 but different distributions.
Incorrect: Replace \(X\) by \(E(X)\) whenever a formula contains \(X\).
Correction: \(X\) is a random variable; \(E(X)\) is a number. Replacing one by the other generally discards variability and changes the quantity being calculated.
An indicator \(I_A\) equals 1 if event \(A\) occurs and 0 otherwise. Its expectation is:
\[E(I_A)=1\cdot P(A)+0\cdot P(A^c)=P(A).\]This is the fundamental bridge: probabilities can be expressed as expectations of indicators. For example, if \(I\) indicates that a die shows six, then \(E(I)=1/6\). If \(J\) indicates at least one Head in two independent fair tosses, then \(E(J)=3/4\).
A count \(C\) can often be written as a sum of indicators, one per counted item. Each indicator asks a simpler yes/no question than the full count distribution.
Core idea: An indicator converts an event into a numerical variable whose expected value is exactly the event’s probability.
For variables with finite expectations and constants \(a,b,c\):
In particular:
\[E(X+Y)=E(X)+E(Y),\qquad E(aX)=aE(X).\]Independence is not required. More generally, for a finite sum:
\[E\left(\sum_{j=1}^nX_j\right)=\sum_{j=1}^nE(X_j).\]On a finite sample space, group outcomes by the value assigned by \(X\) or average directly over individual outcomes:
\[E(X)=\sum_sX(s)P(\{s\}).\]
Both \(X\) and \(Y\) use the same outcome probabilities, so:
\[\begin{aligned} E(X+Y) &=\sum_s[X(s)+Y(s)]P(\{s\})\\ &=\sum_sX(s)P(\{s\})+\sum_sY(s)P(\{s\})\\ &=E(X)+E(Y). \end{aligned}\]No factorization of joint probabilities is used. Even \(Y=X\), an extreme dependence, gives \(E(X+Y)=E(2X)=2E(X)\).
Incorrect: Dependent summands prevent adding expectations.
Correction: Dependence affects the distribution of a sum but does not invalidate linearity. A different operation, such as multiplying variables, does not inherit this rule: generally \(E(XY)\ne E(X)E(Y)\) without additional assumptions.
Let \(X\sim\operatorname{Bin}(n,p)\) count successes in \(n\) independent trials with common probability \(p\). Write:
\[X=I_1+\cdots+I_n,\]where \(I_j\) indicates success on trial \(j\). By the fundamental bridge, \(E(I_j)=p\). Therefore:
\[\boxed{E(X)=np.}\]For ten independent trials with success probability \(0.3\), the expected success count is 3. The observed count remains random; it need not equal 3.
The same answer follows from \(E(X)=\sum_k kP(X=k)\), at the cost of more work:
\[E(X)=\sum_{k=0}^{n}k\binom nk p^kq^{n-k},\qquad q=1-p.\]Without the factor \(k\) this would be the binomial theorem. To remove the \(k\), use the committee-with-president identity \(k\binom nk=n\binom{n-1}{k-1}\). The \(k=0\) term is zero, so start at \(k=1\), and take out \(np\):
\[E(X)=np\sum_{k=1}^{n}\binom{n-1}{k-1}p^{k-1}q^{n-k}.\]With \(j=k-1\), the sum becomes \(\sum_{j=0}^{n-1}\binom{n-1}{j}p^jq^{n-1-j}=(p+q)^{n-1}=1\) by the binomial theorem. Hence \(E(X)=np\).
Core idea: The definition works, but only with the right identity at hand. Linearity with indicators gives the answer in one line, and still works if the indicators are dependent.
The calculation of the mean uses only the individual success probabilities. Independence is needed for the Binomial distribution story, but a count of dependent indicators, each with success probability \(p\), still has mean \(np\) even when its distribution is not Binomial.
Toss one fair coin and copy its result into ten records. The total number of recorded Heads is 0 or 10, each with probability \(1/2\). Its mean is 5, just like \(\operatorname{Bin}(10,1/2)\), but its distribution is very different.
Core idea: Linearity can determine an expected count from marginal probabilities without identifying the joint model or the full count distribution.
Sample \(n\) items uniformly without replacement from a population with \(w\) success-type and \(b\) failure-type items. Let \(X\) count successes and \(I_j\) indicate that draw \(j\) is a success:
\[X=\sum_{j=1}^nI_j.\]By symmetry, each draw position is equally likely to contain any population item. Its unconditional success probability is:
\[P(I_j=1)=\frac{w}{w+b}.\]Earlier draws change conditional probabilities, so the indicators are generally dependent. Nonetheless:
Shuffle a standard deck uniformly and inspect its first five cards. Let \(X\) count aces. The five position indicators each have probability \(4/52=1/13\) of being 1, so:
\[E(X)=5\frac1{13}=\frac5{13}.\]This is an expected count of about 0.385 aces, not a probability of drawing an ace or a claim that a hand contains a fractional card.
An alternative decomposition uses one indicator per ace. Each of the four aces has probability \(5/52\) of appearing in the hand. Thus:
\[E(X)=4\frac5{52}=\frac5{13}.\]Core idea: Choose indicators indexed by draw positions or by population items. Both decompositions count the same quantity, and symmetry supplies the individual probabilities.
Let \(G\sim\operatorname{Geom}(p)\) and \(q=1-p\). By definition:
\[E(G)=\sum_{k=0}^{\infty}kq^kp.\]For \(0<q<1\), differentiate the geometric power series:
\[\sum_{k=0}^{\infty}q^k=\frac1{1-q} \quad\Longrightarrow\quad \sum_{k=1}^{\infty}kq^{k-1}=\frac1{(1-q)^2}.\]Multiplying by \(pq\) gives:
\[E(G)=\frac{pq}{(1-q)^2}=\frac qp.\]Since \(T=G+1\), linearity gives:
For independent fair die rolls until a six, \(E(G)=5\) non-sixes and \(E(T)=6\) total rolls. Neither mean specifies an exact stopping time in a particular experiment.
This argument avoids the series and the derivative; it uses only the story and the fact, shown above, that the mean is finite. Think of flipping a coin with heads probability \(p\) until the first heads, and let \(\mu=E(G)\). A first-trial success contributes zero failures. A first-trial failure contributes one failure, and then the problem starts over: the coin has no memory of the failure, so the remaining wait has the same distribution as before. Averaging these cases:
\[\mu=p\cdot0+q(1+\mu).\]Thus \(p\mu=q\) and \(\mu=q/p\), agreeing with the direct calculation. Independence and unchanged success probability make the restart argument valid.
Return \(n\) labeled items uniformly at random to \(n\) labeled recipients, one item each. Let \(M\) count correct matches and \(I_j\) indicate that recipient \(j\) receives their own item.
Each recipient has probability \(1/n\) of a match by symmetry. Therefore:
\[M=\sum_{j=1}^nI_j,\qquad E(M)=n\frac1n=1.\]The indicators are dependent, but the expected number of matches is 1 for every positive \(n\). This does not mean exactly one match is guaranteed or that the probability of any match is 1.
Let \(X\) equal \(-1\) or 1 with equal probability. Then \(E(X)=0\) but \(X^2=1\) always, so:
\[E(X^2)=1\ne0=(E(X))^2.\]Linearity applies to sums and constant multiples. It does not allow moving expectation through an arbitrary square, reciprocal, or other nonlinear function.
Try these before reading the answers.
Both variables are \(\operatorname{Bern}(1/2)\). They cannot both equal 1, so \(P(X=1,Y=1)=0\), whereas \(P(X=1)P(Y=1)=1/4\). They are dependent.
Here \(p=1/4\) and \(q=3/4\):
\[P(G=2)=\left(\frac34\right)^2\frac14=\frac9{64}.\]The means are \(E(G)=q/p=3\) failures and \(E(T)=1/p=4\) total tosses.
The first three tosses must all fail:
\[P(T\ge4)=q^3=\frac{27}{64}.\]Success on the fourth toss is not required by this event.
The mean is not in the support, so no observation equals it.
Independence is not required; finite expectations suffice here.
Each position has unconditional white probability \(6/10\). Thus \(E(X)=3(6/10)=9/5\). The draw indicators are dependent, but linearity still applies.
The expected total is \(20(0.2)=4\). Dependence may change its distribution, so a Binomial conclusion is not justified.
Each of four recipients matches with probability \(1/4\), giving \(E(M)=1\). This is not the probability of at least one match. For four items, that probability is \(1-9/24=5/8\), using the nine permutations with no matches; its inequality with the mean illustrates the distinction between an event indicator and the match count.
Writing the second result as \(Y=X\):
\[E(X+Y)=E(X)+E(Y)=7.\]The sum takes only the even values \(2,4,6,8,10,12\), each with probability \(1/6\). Its mean equals the independent-dice mean, but its distribution does not.
| Concept | Essential fact |
|---|---|
| Independence, discrete case | Joint value probabilities factor for every pair |
| Identical distributions | Does not imply independence or equal observed values |
| Geometric convention | \(G\) counts failures before success, starting at 0 |
| Geometric PMF | \(P(G=k)=q^kp\) |
| Total waiting time | \(T=G+1\) counts the successful trial |
| Expectation | \(E(X)=\sum_xxP(X=x)\) |
| Interpretation | Probability-weighted average; need not be an attainable value |
| Fundamental bridge | \(E(I_A)=P(A)\) |
| Linearity | \(E(\sum_j a_jX_j+c)=\sum_j a_jE(X_j)+c\) for finite expectations |
| Dependence | Does not invalidate linearity |
| Binomial mean | \(np\), by indicators; directly via \(k\binom nk=n\binom{n-1}{k-1}\) and the binomial theorem |
| Hypergeometric mean | \(nw/(w+b)\) |
| Geometric mean | \((1-p)/p\) failures, by differentiating a series or by restarting: \(\mu=p\cdot0+q(1+\mu)\) |
| First Success mean | \(1/p\) total trials |
| Nonlinear transformations | Generally \(E(g(X))\ne g(E(X))\) |
Variables whose joint probabilities factor into marginal probabilities, so observing one does not change the distribution of the other.
The distribution of failures before the first success in independent trials with a fixed success probability, under the convention used here.
The distribution of the total number of trials up to and including the first success; one more than the corresponding failure count.
A probability-weighted average of a random variable’s values, also called its mean or expectation.
The identity \(E(I_A)=P(A)\) connecting event probabilities with expectations of their indicators.
Finite sums and constant multiples pass through expectation when the expectations are finite, regardless of dependence.
Invariance of the probability model under exchanging equivalent labels or positions, allowing their event probabilities to be identified as equal.
Linearity follows from averaging different numerical functions over the same experimental outcomes. Combined with indicators and symmetry, it makes the average number of local maxima in a permutation easy to calculate despite complicated dependence. Waiting for several successes gives the Negative Binomial distribution, a sum of Geometric failure counts. The St. Petersburg game shows a different limit of expectation: a payoff can be finite in every completed game while its mean is infinite.
| Symbol | Meaning |
|---|---|
| \(E(X)\) | Expected value of random variable \(X\) |
| \(s\in S\) | Outcome in a sample space |
| \(a_1,\ldots,a_n\) | A permutation of \(1,\ldots,n\) |
| \(I_j\) | Indicator of a local maximum at position \(j\) |
| \(L=\sum_jI_j\) | Number of local maxima |
| \(p\), \(q=1-p\) | Success and failure probabilities in independent trials |
| \(r\) | Fixed positive integer number of successes being awaited |
| \(X\sim\operatorname{NBin}(r,p)\) | Failures before the \(r\)th success |
| \(G_j\) | Failures between the \((j-1)\)st and \(j\)th successes |
| \(T=X+r\) | Total trials through the \(r\)th success |
| \(N\) | Toss number of the first Head in St. Petersburg |
| \(W=2^N\) | Ideal St. Petersburg payoff |
| \(m\) | Cutoff toss number in a finite-payoff version |
After studying these notes, you should be able to:
For a finite sample space, the PMF definition of expectation can be rewritten as:
\[E(X)=\sum_x xP(X=x)=\sum_{s\in S}X(s)P(\{s\}).\]To obtain the second expression, replace each \(P(X=x)\) by the sum of probabilities of outcomes mapped to \(x\). Every outcome contributes its assigned value times its probability.
Now \(X\) and \(Y\) use the same outcome weights:
\[\begin{aligned} E(X+Y) &=\sum_s(X(s)+Y(s))P(\{s\})\\ &=\sum_sX(s)P(\{s\})+\sum_sY(s)P(\{s\})\\ &=E(X)+E(Y). \end{aligned}\]Similarly, \(E(cX)=cE(X)\) for a constant \(c\). The same conclusion extends to general random variables with finite expectations; the finite-space argument makes its arithmetic structure explicit.
No independence assumption appears. Dependence may change the distribution of the sum, but not this identity.
Core idea: Average the sum at each outcome, or add the separate averages. Both use the same probabilities and differ only in the order of addition.
Choose a permutation \(a_1,\ldots,a_n\) of \(1,\ldots,n\) uniformly, with all \(n!\) permutations equally likely. Assume \(n\ge2\).
An interior position \(j\) is a local maximum when:
\[a_j>a_{j-1}\quad\text{and}\quad a_j>a_{j+1}.\]At the endpoints, there is only one neighbor: position 1 is a local maximum if \(a_1>a_2\), and position \(n\) is one if \(a_n>a_{n-1}\).
For example, \(4,2,5,3,6,1\) has local maxima at positions 1, 3, and 5. A local maximum need not be the largest value in the whole permutation.
The average count across all permutations is an expectation. Let \(I_j\) indicate a local maximum at position \(j\), so \(L=\sum_{j=1}^nI_j\).
For \(2\le j\le n-1\), the three values at \(j-1,j,j+1\) are distinct. Their relative ordering is uniform among the six possible orders. Each position is equally likely to contain the largest of the three, so:
\[E(I_j)=P(I_j=1)=\frac13.\]Each endpoint is equally likely to exceed or fall below its sole neighbor:
\[E(I_1)=E(I_n)=\frac12.\]By linearity:
For \(n=6\), the expected count is \(7/3\), while the particular illustrated permutation has three maxima. For \(n=2\), exactly one of the two positions is a maximum, consistent with the formula.
Incorrect: All positions have probability \(1/3\) of being maxima, or the indicators must be independent to add their expectations.
Correction: Endpoints have only one neighbor and probability \(1/2\). Adjacent positions cannot both be strict local maxima, so their indicators are dependent; linearity still applies.
Core idea: The global count has a complicated distribution, but each local indicator requires only the relative order of two or three values.
Run independent Bernoulli trials, each with the same success probability \(p\), where \(0<p<1\). Stop as soon as the \(r\)th success occurs. Let \(X\) count failures before that success:
\[X\sim\operatorname{NBin}(r,p),\qquad X\in\{0,1,2,\ldots\}.\]The stopping rule differs from Binomial sampling:
| Model | Fixed in advance | Random quantity counted |
|---|---|---|
| Binomial | Number of trials \(n\) | Successes in those trials |
| Geometric | One required success | Failures before that success |
| Negative Binomial | Number of successes \(r\) | Failures before the \(r\)th success |
When \(r=1\), the Negative Binomial is the Geometric distribution under the failures-before-success convention.
If \(X=k\), there were \(k\) failures and \(r\) successes, so the experiment lasted \(k+r\) trials. Its final trial must be the \(r\)th success. In the preceding \(k+r-1\) positions, choose where the other \(r-1\) successes occurred:
\[\binom{k+r-1}{r-1}.\]Each admissible sequence has probability \(p^rq^k\). Adding their probabilities gives:
The coefficient can equivalently be written \(\binom{k+r-1}{k}\). The distribution is normalized because independent trials with positive success probability eventually accumulate any fixed number \(r\) of successes with probability 1.
Toss a fair coin independently until the third Head. Exactly two preceding Tails means five tosses total, with the last toss Heads. Among the first four positions, choose the two positions containing the earlier Heads:
\[P(X=2)=\binom42\left(\frac12\right)^3\left(\frac12\right)^2 =\frac6{32}=\frac3{16}.\]The six admissible sequences end in Heads. Using \(\binom53\) would also count five-toss sequences whose third Head occurred earlier, after which this experiment would already have stopped.
Let \(G_1\) count failures before the first success, \(G_2\) failures between the first and second successes, and so on through \(G_r\). Then:
\[X=G_1+\cdots+G_r.\]Every interval restarts independent trials with the same success probability, so the \(G_j\) are i.i.d. \(\operatorname{Geom}(p)\) variables. Since \(E(G_j)=q/p\):
\[\boxed{E(X)=\frac{rq}{p}.}\]The total trial count \(T=X+r\) includes the \(r\) successful trials:
\[\boxed{E(T)=\frac{rq}{p}+r=\frac rp.}\]Its PMF is obtained by shifting the failure count:
\[P(T=t)=\binom{t-1}{r-1}p^rq^{t-r},\qquad t=r,r+1,\ldots.\]For the third-Head example, \(E(X)=3\) Tails and \(E(T)=6\) total tosses.
Incorrect: Every source’s “Negative Binomial” variable starts at 0 and has mean \(rq/p\).
Correction: Some sources count failures, while others count all trials. Define the variable explicitly. Counting all trials shifts the support to \(\{r,r+1,\ldots\}\) and the mean to \(r/p\).
Core idea: A waiting time for several successes is a sum of waiting intervals. Linearity obtains its mean without summing the Negative Binomial PMF.
Toss a fair coin independently until the first Head. If that Head appears on toss \(N=n\), receive a payoff of \(2^n\) dollars. Thus:
\[P(N=n)=2^{-n},\qquad n=1,2,3,\ldots,\]and \(W=2^N\). The payoff distribution begins:
| First Head on toss \(n\) | Probability | Payoff | Contribution to \(E(W)\) |
|---|---|---|---|
| 1 | \(1/2\) | 2 | 1 |
| 2 | \(1/4\) | 4 | 1 |
| 3 | \(1/8\) | 8 | 1 |
| \(n\) | \(2^{-n}\) | \(2^n\) | 1 |
Every toss number contributes one dollar to the expectation:
The game ends with probability 1 because \(P(N>m)=2^{-m}\to0\). Its payout is finite whenever it ends. Infinite expectation therefore does not mean infinite winnings in a typical game, or any positive probability of an infinite payoff. It means the weighted tail cannot be summed to a finite mean.
The paradox is that this idealized infinite expected payoff does not correspond to an unlimited practical willingness to pay. The mathematical model permits arbitrarily large payouts, while a real payer has finite resources. Expected money alone also does not describe a person’s preferences over uncertain outcomes.
Core idea: Exponentially small probabilities can be exactly offset by exponentially large payoffs, making the total expected contribution diverge.
Keep the original payout for \(N\le m\), but pay nothing when \(N>m\). Let:
\[W_m=2^N I_{\{N\le m\}}.\]Only the first \(m\) one-dollar contributions remain:
\[\boxed{E(W_m)=\sum_{n=1}^m2^n2^{-n}=m.}\]With \(m=40\), this expectation is $40, even though the probability of crossing the cutoff is only \(2^{-40}\).
Instead define \(V_m=\min(2^N,2^m)\). Retain the original payout through toss \(m\), and pay the cap if the first Head arrives later. Then:
\[\begin{aligned} E(V_m) &=\sum_{n=1}^m2^n2^{-n}+2^mP(N>m)\\ &=m+2^m2^{-m}=m+1. \end{aligned}\]With \(m=40\), the capped expectation is $41. Both tail rules make the mean finite, but they define different payoff variables.
Incorrect: An extremely rare cutoff can have only a negligible effect on the expected payoff.
Correction: Rarity alone does not determine an event’s contribution to expectation. Its payoff matters too. The omitted infinite tail contributes an infinite amount in the ideal game despite having very small probability.
The first-Head trial count has \(E(N)=2\). But:
\[E(2^N)=\infty,\qquad 2^{E(N)}=4.\]Replacing a random input by its mean before applying a nonlinear function can give an entirely different result. To calculate an expected payoff, weight the actual payoff in each case by that case’s probability.
If \(X\) takes 0 and 2 with equal probability, then \(E(X)=1\). Yet:
\[E(X^2)=\frac12(0^2)+\frac12(2^2)=2, \qquad (E(X))^2=1.\]The failure of \(E(g(X))=g(E(X))\) is not restricted to infinite means. Linearity applies to sums and constant multiples, not arbitrary nonlinear transformations.
Try these before reading the answers.
Both variables are averaged over the same outcomes with the same probabilities. Distributing the weighted sum over \(X(s)+Y(s)\) proves the identity without factoring any joint probabilities.
With endpoints, \(E(L)=(8+1)/3=3\). Counting only interiors gives \((8-2)/3=2\). The two endpoints contribute \(1/2\) each.
Positions 1 and 3 are maxima: 3 exceeds its sole neighbor 1, and 4 exceeds both neighbors 1 and 2. The count is 2.
With \(r=2\), \(k=3\), \(p=1/3\), and \(q=2/3\):
\[P(X=3)=\binom41\left(\frac13\right)^2\left(\frac23\right)^3 =\frac{32}{243}.\]The last of five trials is a success; the other success can occupy any of the first four positions.
The total count exceeds the failure count by the two required successful trials.
That coefficient counts all sequences with \(r\) successes in \(k+r\) positions, including sequences whose \(r\)th success occurs before the last position. The experiment would stop earlier in those cases. Fix the final success and choose only \(r-1\) earlier success positions, giving \(\binom{k+r-1}{r-1}\).
\(W\ge16\) is equivalent to \(N\ge4\), which requires Tails on the first three tosses:
\[P(W\ge16)=\frac18.\]The game ends almost surely and its realized payout is finite almost surely. Its infinite mean comes from the unbounded weighted tail.
The zero-tail version has mean $5. The capped version has mean $6, since the tail adds \(2^5P(N>5)=32/32=1\) dollar to the expectation.
Exponentiation is nonlinear. The expected payoff is \(\sum_n2^nP(N=n)\), which diverges, whereas exponentiating the mean trial count gives 4. They are different operations on the distribution.
| Concept | Essential fact |
|---|---|
| Linearity proof | Average over common outcomes and distribute the sum |
| Independence | Not needed for linearity with finite expectations |
| Interior local maximum | Largest of three neighboring values: probability \(1/3\) |
| Endpoint maximum | Larger than its sole neighbor: probability \(1/2\) |
| Mean maxima count | \((n+1)/3\) for \(n\ge2\), including endpoints |
| Negative Binomial story | Failures before a fixed \(r\)th success |
| PMF | \(\binom{k+r-1}{r-1}p^r(1-p)^k\), \(k\ge0\) |
| Geometric decomposition | \(X=\sum_{j=1}^rG_j\) with i.i.d. \(\operatorname{Geom}(p)\) intervals |
| Failure-count mean | \(rq/p\) |
| Total-trial mean | \(r/p\) |
| St. Petersburg | \(P(N=n)=2^{-n}\), payoff \(2^n\), mean infinite |
| Zero payoff beyond \(m\) | Mean \(m\) |
| Payout capped at \(2^m\) | Mean \(m+1\) |
| Nonlinear expectation | Generally \(E(g(X))\ne g(E(X))\) |
A permutation value larger than each immediate neighbor, with only one comparison at an endpoint under the convention used here.
The distribution of failures before a fixed number of successes in independent Bernoulli trials with a common success probability.
The condition determining when an experiment ends, such as reaching the \(r\)th success. It constrains which outcome sequences are admissible.
For a nonnegative variable, divergence of the probability-weighted sum of values, even when the realized value is finite almost surely.
The part of an expected value contributed by outcomes beyond a specified threshold. Both their probabilities and values determine its size.
A payoff limited to a fixed maximum, represented here by \(\min(2^N,2^m)\).
A random variable and its distribution are different objects. Arithmetic on random variables does not translate into the same arithmetic on their PMFs. The Poisson distribution counts rare events across many opportunities. It has one parameter, the rate \(\lambda\), which is also its mean. The Poisson paradigm approximates the number of events that occur among many unlikely, independent or weakly dependent events by a Poisson count with the same mean. The Binomial distribution converges to the Poisson when the number of trials grows and the success probability shrinks with their product held fixed. Birthday problems show the approximation in use: matching pairs and matching triples become one-line calculations.
| Symbol | Meaning |
|---|---|
| \(X\sim\operatorname{Pois}(\lambda)\) | \(X\) has the Poisson distribution with rate \(\lambda>0\) |
| \(\lambda\) | Rate parameter: the expected number of events |
| \(e^{-\lambda}\lambda^k/k!\) | Poisson probability of exactly \(k\) events |
| \(A_1,\ldots,A_n\) | Events whose occurrences are counted |
| \(p_j=P(A_j)\) | Probability of the \(j\)th event |
| \(I(A_j)\) | Indicator of event \(A_j\) |
| \(X=\sum_jI(A_j)\) | Number of the events \(A_j\) that occur |
| \(\operatorname{Bin}(n,p)\) | Successes in \(n\) independent trials with success probability \(p\) |
| \(q=1-p\) | Failure probability |
| \(I_{ijk}\) | Indicator that persons \(i<j<k\) share a birthday |
| \(\approx\) | Approximately equal (an approximation, not an identity) |
After studying these notes, you should be able to:
A random variable \(X\) is a function that assigns a number to each outcome of an experiment. Its distribution, for example its PMF \(p_X(x)=P(X=x)\), records how probability is spread over the possible values. The distribution describes the random variable, but it is not the random variable. Treating one as the other is called sympathetic magic: manipulating the description as if that manipulated the object itself.
The distribution is a blueprint and the random variable is a house built from it. Many houses can share one blueprint. Likewise, many random variables, possibly defined on different experiments, can share one distribution.
Let \(X\) and \(Y\) be random variables with PMFs \(p_X\) and \(p_Y\). A tempting but wrong claim is
\[P(X+Y=t)\overset{?}{=}P(X=t)+P(Y=t).\]The right-hand side need not even be a valid probability. Take a fair coin, with \(X\) the indicator of Heads and \(Y=X\). Then \(P(X=1)+P(Y=1)=1/2+1/2=1\), and summing the claimed PMF over \(t\in\{0,1\}\) gives a total of 2, not 1.
The correct calculation starts from the random variables. Here \(X+Y=2X\) takes the value 0 or 2, each with probability \(1/2\):
\[P(X+Y=0)=\frac12,\qquad P(X+Y=2)=\frac12.\]The value 1 is impossible, although the incorrect formula assigns it probability 1.
The same error appears when scaling. The PMF of \(2X\) moves each value \(x\) to \(2x\) while keeping its probability. That is a horizontal stretch of the PMF. Multiplying the PMF of \(X\) by 2 is a vertical stretch, and the result sums to 2.
A second form of the error is to conclude from equal distributions that the random variables are equal. In the coin example, the indicator of Heads \(X\) and the indicator of Tails \(1-X\) both have the \(\operatorname{Bern}(1/2)\) distribution, yet \(X=1-X\) never happens.
Incorrect: If \(X\) and \(Y\) have the same distribution, then \(X=Y\), or at least \(P(X=Y)\) is large.
Correction: Equal distributions only mean equal probabilities for each value. Random variables with the same distribution can be equal always, sometimes, or never.
Core idea: Before writing an answer, check its category: a number, an event, a random variable, or a distribution. Many errors in this course are category errors.
A random variable \(X\) has the Poisson distribution with parameter \(\lambda>0\), written \(X\sim\operatorname{Pois}(\lambda)\), if
\[P(X=k)=\frac{e^{-\lambda}\lambda^k}{k!},\qquad k=0,1,2,\ldots.\]The support is all nonnegative integers. Unlike the Binomial, there is no upper bound on the count.
Each term is positive because \(e^{-\lambda}>0\), \(\lambda^k>0\), and \(k!>0\). For the total, use the Taylor series of the exponential function, which converges for every real \(\lambda\):
\[e^{\lambda}=\sum_{k=0}^{\infty}\frac{\lambda^k}{k!}.\]Factor \(e^{-\lambda}\) out of the sum:
The factor \(e^{-\lambda}\) is exactly the normalizing constant that turns the terms of the exponential series into probabilities.
The ratio of consecutive probabilities is
\[\frac{P(X=k+1)}{P(X=k)}=\frac{\lambda}{k+1}.\]The probabilities increase while \(k+1<\lambda\) and decrease once \(k+1>\lambda\), so the PMF rises to a peak near \(\lambda\) and then falls. For small \(\lambda\) the PMF is strongly right-skewed. For larger \(\lambda\) it becomes more symmetric and bell-shaped.
Let \(X\sim\operatorname{Pois}(\lambda)\). By definition of expectation,
\[E(X)=\sum_{k=0}^{\infty}k\frac{e^{-\lambda}\lambda^k}{k!}.\]The derivation has three steps.
The parameter is therefore the average count. If a count is modeled as Poisson, \(\lambda\) is read as a rate: the expected number of events per hour, per page, per cookie, or per region.
Textbook Example 4.7.2 shows that \(\operatorname{Var}(X)=\lambda\) as well. A short route uses the same shifting idea on \(E(X(X-1))\):
\[E(X(X-1))=e^{-\lambda}\sum_{k=2}^{\infty}\frac{\lambda^k}{(k-2)!}=\lambda^2.\]Then \(E(X^2)=E(X(X-1))+E(X)=\lambda^2+\lambda\), so \(\operatorname{Var}(X)=E(X^2)-(E(X))^2=\lambda\). Equality of mean and variance is a quick diagnostic: count data whose variance is far larger than their mean are poorly described by a single Poisson model.
The Poisson is used for the number of successes when there are many opportunities for success, each with a small probability. Typical counts are:
| Count | Many opportunities | Small probability for each |
|---|---|---|
| Emails received in an hour | Many possible senders, or many short time slots | Any given sender writes, or any given millisecond brings an email |
| Chocolate chips in a cookie | Many small cubes of dough | Any given cube contains a chip |
| Earthquakes in a region in a year | Many places and times | Any given place and time has a quake |
| Buses arriving at a stop in a day | Many short intervals | Any given interval contains an arrival |
The counts above are not exactly Poisson. The number of possible senders is finite, and only so many chips fit in a cookie, while a Poisson variable is unbounded. The Poisson is an approximation, and the approximation is good when the individual probabilities are small.
Incorrect: The Poisson “law of rare events” requires the total count to be small, so \(\lambda\) must be small.
Correction: “Rare” refers to each individual event \(A_j\) having a small probability. The total rate \(\lambda=\sum_jp_j\) can be large, because many small probabilities add up. A popular inbox can have \(\lambda=20\) emails per hour even though any single person is unlikely to write in that hour.
Let \(A_1,\ldots,A_n\) be events with \(p_j=P(A_j)\). Suppose \(n\) is large, every \(p_j\) is small, and the events are independent or weakly dependent. Let
\[X=\sum_{j=1}^nI(A_j)\]count how many of the events occur. Then
\[X\ \dot\sim\ \operatorname{Pois}(\lambda),\qquad \lambda=\sum_{j=1}^np_j.\]Here \(\dot\sim\) means “is approximately distributed as.” Three features make the paradigm flexible:
The approximation is most useful for small counts, especially for
\[P(X\ge1)=1-P(X=0)\approx1-e^{-\lambda}.\]Computing \(P(X=0)\) exactly requires the joint behavior of all the events. The approximation requires only their total probability.
For independent \(A_j\) and \(N\sim\operatorname{Pois}(\lambda)\), the textbook (Section 4.7) quotes a bound, proved by the Stein–Chen method: for every set \(B\) of nonnegative integers,
\[|P(X\in B)-P(N\in B)|\le\min\left(1,\frac1\lambda\right)\sum_{j=1}^np_j^2.\]The error is controlled by \(\sum_jp_j^2\). This makes “small \(p_j\)” precise: the squares of the probabilities must sum to something small compared with \(\lambda\). For instance, if all \(p_j=p\), the right side is at most \(\min(np^2,p)\).
The Binomial is the special case of the Poisson paradigm in which the events are independent and share one probability \(p\). In that case the approximation can be proved exactly as a limit.
Let \(X\sim\operatorname{Bin}(n,p)\). If \(n\to\infty\) and \(p\to0\) with \(\lambda=np\) held fixed, then for each fixed \(k\in\{0,1,2,\ldots\}\),
\[P(X=k)\to\frac{e^{-\lambda}\lambda^k}{k!}.\]Proof. Fix \(k\) and take \(n\ge k\). Substitute \(p=\lambda/n\) into the Binomial PMF and expand the binomial coefficient:
\[\begin{aligned} P(X=k) &=\binom nk\left(\frac{\lambda}{n}\right)^k\left(1-\frac{\lambda}{n}\right)^{n-k}\\ &=\frac{\lambda^k}{k!}\cdot\frac{n(n-1)\cdots(n-k+1)}{n^k}\cdot\left(1-\frac{\lambda}{n}\right)^{n}\cdot\left(1-\frac{\lambda}{n}\right)^{-k}. \end{aligned}\]Take the limit of each factor separately as \(n\to\infty\).
| Factor | Limit | Reason |
|---|---|---|
| \(\dfrac{\lambda^k}{k!}\) | \(\dfrac{\lambda^k}{k!}\) | Does not depend on \(n\) |
| \(\dfrac{n(n-1)\cdots(n-k+1)}{n^k}=\prod_{i=0}^{k-1}\left(1-\dfrac in\right)\) | \(1\) | A fixed number \(k\) of factors, each tending to 1 |
| \(\left(1-\dfrac{\lambda}{n}\right)^n\) | \(e^{-\lambda}\) | Compound-interest limit \(\left(1+\dfrac xn\right)^n\to e^x\) with \(x=-\lambda\) |
| \(\left(1-\dfrac{\lambda}{n}\right)^{-k}\) | \(1\) | Fixed power of a quantity tending to 1 |
Multiplying the limits gives
The key step is that \(k\) stays fixed while \(n\) grows. A fixed number of factors near 1 multiplies to something near 1. The only factor whose exponent grows with \(n\) is \((1-\lambda/n)^n\), and it produces the \(e^{-\lambda}\).
Incorrect: Any Binomial with large \(n\) is approximately Poisson.
Correction: The success probability must be small. If \(p\) stays fixed while \(n\) grows, the mean \(np\) and the variance \(np(1-p)\) grow without bound, and the variance stays below the mean. That is a different regime (later approximated by the Normal). The Poisson limit needs \(p\to0\) with \(np\) moderate.
A machine produces 1000 items independently, and each is defective with probability \(0.003\). Find the probability of at most one defect.
Model. The defect count is \(X\sim\operatorname{Bin}(1000,0.003)\). Here \(n\) is large, \(p\) is small, and \(\lambda=np=3\) is moderate, so use \(\operatorname{Pois}(3)\).
Approximation.
\[P(X\le1)\approx e^{-3}+e^{-3}\cdot3=4e^{-3}\approx0.1991.\]Exact value. \(P(X\le1)=(0.997)^{1000}+1000(0.003)(0.997)^{999}\approx0.1987\).
The absolute error is about \(0.0004\), well within the bound \(\min(p,np^2)=0.003\).
Count the raindrops that land on a patch of ground during one minute. To fit a Binomial model, divide the patch into a fine grid of \(n\) tiny squares and call a square a success if a drop lands in it. Each square has a small chance of a hit, and there are very many squares.
The Binomial model has three weaknesses here:
Computing with \(\binom nk\) for astronomically large \(n\) is also numerically awkward. The Poisson PMF needs only \(\lambda\), the expected number of drops.
Core idea: The Poisson is the natural model when events can happen anywhere in a continuum of time or space and any small piece rarely contains one.
Assume 365 equally likely birthdays, independent across people, and ignore February 29.
With \(m\) people there are \(\binom m2\) pairs. Let \(A_{ij}\) be the event that persons \(i<j\) share a birthday. Each has probability \(1/365\). The events are weakly dependent: for example, if persons 1 and 2 match and persons 2 and 3 match, then persons 1 and 3 must match. The Poisson paradigm gives
\[X=\#\{\text{matching pairs}\}\ \dot\sim\ \operatorname{Pois}(\lambda),\qquad \lambda=\binom m2\frac1{365}.\]For \(m=23\), there are \(\binom{23}2=253\) pairs, so \(\lambda=253/365\approx0.6932\). Then
\[P(\text{at least one match})=1-P(X=0)\approx1-e^{-253/365}\approx0.5000.\]The exact answer from Lecture 3 is about \(0.5073\), so the approximation recovers the familiar threshold of 23 people.
Textbook Example 4.7.5 asks for at least one pair with birthdays at most one day apart. For two given people, the second must have birthday on the first person’s day or on one of the two adjacent days, so the probability is \(3/365\). Then \(\lambda=\binom m2\frac3{365}\). Setting \(1-e^{-\lambda}\ge1/2\) requires \(\lambda\ge\ln2\), which first holds at \(m=14\). The exact answer is also 14, while an exact calculation without the approximation is much harder.
Now ask for three people who share a birthday. For each triple \(i<j<k\), let \(I_{ijk}\) indicate that all three share a birthday. Whatever the birthday of person \(i\), persons \(j\) and \(k\) must both match it:
\[E(I_{ijk})=P(\text{triple }ijk\text{ matches})=\left(\frac1{365}\right)^2.\]By linearity, the expected number of triple matches among \(m\) people is
\[E\left(\sum_{i<j<k}I_{ijk}\right)=\binom m3\frac1{365^2}.\]The indicators are dependent. For example, \(I_{123}\) and \(I_{124}\) share persons 1 and 2, so knowing \(I_{123}=1\) raises the chance that \(I_{124}=1\). But each triple is very unlikely and there are many triples, so the dependence is weak. Hence, with \(\lambda=\binom m3/365^2\),
Require \(1-e^{-\lambda}\ge1/2\), that is, \(\lambda\ge\ln2\approx0.6931\). This means
\[\binom m3\ge0.6931\times365^2\approx92{,}340.\]Since \(\binom m3\approx m^3/6\), this gives \(m\approx\sqrt[3]{554{,}000}\approx82\). Checking exact values of \(\binom m3\):
| \(m\) | \(\lambda=\binom m3/365^2\) | \(1-e^{-\lambda}\) |
|---|---|---|
| 82 | 0.665 | 0.486 |
| 83 | 0.690 | 0.498 |
| 84 | 0.715 | 0.511 |
The approximation therefore suggests about 84 people.
Supplementary caveat. The exact threshold for a triple match is 88 people. The approximation is less accurate here than for pairs because the dependence is stronger. A single day shared by four people creates \(\binom43=4\) matching triples at once, so matching triples arrive in clumps. Clumping raises \(P(X=0)\) above \(e^{-\lambda}\) for a given mean, so the Poisson estimate is slightly too optimistic.
Incorrect: The Poisson paradigm needs independent events, so it cannot be used for birthday matches.
Correction: Weak dependence is allowed. Pair and triple indicators overlap in people and are dependent, but each event is rare and any one event’s dependence on the rest is slight. The approximation is excellent for pairs and reasonable for triples.
Try these before reading the answers.
Since \(Y=1-X\), the sum is \(X+Y=1\) for every outcome, so \(P(X+Y=1)=1\). The proposed formula assigns \(p_X(1)+p_Y(1)=1/6+5/6=1\) to the value 1 and \(p_X(0)+p_Y(0)=5/6+1/6=1\) to the value 0, a total of 2. It treats PMFs as if they were the random variables. The distribution of a sum depends on how \(X\) and \(Y\) are related, not only on their separate PMFs.
\(P(X=0)=e^{-\lambda}=0.1\) gives \(\lambda=\ln10\approx2.303\).
From \(P(X=k+1)/P(X=k)=\lambda/(k+1)\), the PMF increases from \(k\) to \(k+1\) exactly when \(k+1<\lambda\) and decreases when \(k+1>\lambda\). If \(\lambda\) is not an integer, the last increase reaches \(k=\lfloor\lambda\rfloor\), after which the PMF decreases. (If \(\lambda\) is an integer, \(\lambda-1\) and \(\lambda\) tie.)
Here \(\lambda=300\times0.005=1.5\):
\[P(X\ge2)=1-e^{-1.5}(1+1.5)=1-2.5e^{-1.5}\approx0.4422.\]\(\lambda=\binom{30}2/365=435/365\approx1.1918\), so \(1-e^{-\lambda}\approx0.6963\). The approximation is about \(0.010\) below the exact \(0.7063\).
\(\lambda=\binom{100}3/365^2=161{,}700/133{,}225\approx1.214\), so \(P\approx1-e^{-1.214}\approx0.703\). The clumping caveat in Part 8 suggests the true value is somewhat smaller.
With \(p\) fixed, \(E(X)=np\to\infty\), so for each fixed \(k\), \(P(X=k)\to0\). Probability escapes to ever larger values and no limiting PMF exists. A Poisson limit requires the expected count \(np\) to stay bounded, which forces \(p\to0\).
The ratio is \(\prod_{i=0}^{k-1}(1-i/n)\). With \(k\) fixed, it is a product of finitely many factors, each tending to 1. If \(k\) grew with \(n\), for example \(k=n\), the product would become \(n!/n^n\), which tends to 0.
| Concept | Essential fact |
|---|---|
| Sympathetic magic | Confusing a random variable with its distribution |
| Sum of random variables | Find its distribution from the variables, not by adding PMFs |
| Scaling | PMF of \(2X\) relocates values; \(2p_X\) is not a PMF |
| Poisson PMF | \(P(X=k)=e^{-\lambda}\lambda^k/k!\), \(k=0,1,2,\ldots\) |
| Validity | Taylor series \(\sum_k\lambda^k/k!=e^\lambda\) |
| Mean | \(E(X)=\lambda\) by dropping \(k=0\) and re-indexing |
| Variance (supplementary) | \(\operatorname{Var}(X)=\lambda\) |
| Consecutive ratio | \(P(X=k+1)/P(X=k)=\lambda/(k+1)\) |
| Poisson paradigm | Count of many rare, weakly dependent events \(\dot\sim\operatorname{Pois}(\sum_jp_j)\) |
| “Rare” | Each \(p_j\) small; \(\lambda\) itself need not be small |
| Binomial limit | \(\operatorname{Bin}(n,\lambda/n)\to\operatorname{Pois}(\lambda)\) as \(n\to\infty\) |
| Key limit | \((1-\lambda/n)^n\to e^{-\lambda}\) |
| At least one event | \(P(X\ge1)\approx1-e^{-\lambda}\) |
| Birthday pairs | \(\lambda=\binom m2/365\); 23 people gives about 0.50 |
| Triple matches | \(\lambda=\binom m3/365^2\); about 84 people by approximation, 88 exactly |
The category error of confusing a random variable with its distribution, for example by adding or scaling PMFs to represent adding or scaling random variables.
An answer of the wrong type of mathematical object, such as a distribution where a number is required. It is necessarily wrong, not merely inaccurate.
The distribution on \(\{0,1,2,\ldots\}\) with PMF \(e^{-\lambda}\lambda^k/k!\) for a rate \(\lambda>0\). Its mean and variance both equal \(\lambda\).
The Poisson parameter, interpreted as the expected number of events in the given unit of time, area, or volume.
The approximation of the number of occurring events among many independent or weakly dependent rare events by a Poisson distribution with \(\lambda=\sum_jP(A_j)\). Also called the law of rare events.
Dependence in which learning whether some of the events occurred changes the probabilities of the others only slightly.
As \(n\to\infty\) and \(p\to0\) with \(np=\lambda\) fixed, the \(\operatorname{Bin}(n,p)\) PMF converges pointwise to the \(\operatorname{Pois}(\lambda)\) PMF.
Data whose values are nonnegative integers, such as numbers of arrivals or occurrences. The Poisson is a standard first model for such data.
A continuous random variable can take any value in an interval, and every single value has probability 0. Its distribution is described by a probability density function (PDF), whose integral over a region gives the probability of that region. The PDF is the derivative of the CDF, and the CDF is the accumulated area under the PDF. Expectation replaces sums with integrals. Variance measures spread as the average squared distance from the mean, and the standard deviation returns it to the original units. LOTUS computes \(E(g(X))\) directly from the distribution of \(X\). The Uniform distribution, a completely random point in an interval, is the simplest continuous model: probability is proportional to length.
| Symbol | Meaning |
|---|---|
| \(F(x)=P(X\le x)\) | Cumulative distribution function (CDF) |
| \(f(x)\) | Probability density function (PDF) of a continuous random variable |
| \(F'(x)=f(x)\) | The PDF is the derivative of the CDF |
| \(\int_a^bf(x)\,dx\) | Probability that \(X\) lies between \(a\) and \(b\) |
| \(\mu=E(X)\) | Mean of \(X\) |
| \(\operatorname{Var}(X)=E\left((X-\mu)^2\right)\) | Variance of \(X\) |
| \(\operatorname{SD}(X)=\sqrt{\operatorname{Var}(X)}\) | Standard deviation of \(X\) |
| \(EX^2\) | Shorthand for \(E(X^2)\), not \((EX)^2\) |
| \(g(X)\) | A function of the random variable \(X\) |
| \(U\sim\operatorname{Unif}(a,b)\) | \(U\) is Uniform on the interval \((a,b)\) |
| \(\varepsilon\) | Length of a small interval |
After studying these notes, you should be able to:
A discrete random variable takes values in a list, finite or countably infinite, such as \(\{0,1,2,\ldots\}\). A continuous random variable takes values in an interval of the real line, such as \((0,1)\) or \((-\infty,\infty)\). The two cases share the CDF but differ in how probability is assigned to individual values.
| Object | Discrete random variable | Continuous random variable |
|---|---|---|
| CDF | \(F(x)=P(X\le x)\), a step function | \(F(x)=P(X\le x)\), continuous, differentiable except at finitely many points |
| Mass at one point | \(P(X=x)\) is the PMF, the jump of \(F\) at \(x\) | \(P(X=x)=0\) for every \(x\) |
| Description | PMF, nonnegative and summing to 1 | PDF \(f=F'\), nonnegative and integrating to 1 |
| Probability of a set | Sum the PMF over the set | Integrate the PDF over the set |
| Expectation | \(\sum_xxP(X=x)\) | \(\int_{-\infty}^{\infty}xf(x)\,dx\) |
| LOTUS | \(\sum_xg(x)P(X=x)\) | \(\int_{-\infty}^{\infty}g(x)f(x)\,dx\) |
The rule of translation is simple: sums become integrals and the PMF becomes the PDF.
A random variable \(X\) is continuous if its CDF \(F\) is differentiable, allowing finitely many points (such as endpoints of the support) where \(F\) is continuous but not differentiable. Its probability density function is
\[f(x)=F'(x).\]The support of \(X\) is the set where \(f(x)>0\).
Conversely, every function with these two properties is the PDF of some continuous random variable.
For a continuous \(X\) and any number \(a\),
\[P(X=a)\le P(a-\varepsilon<X\le a)=F(a)-F(a-\varepsilon)\to0\quad\text{as }\varepsilon\to0,\]because \(F\) is continuous. Hence \(P(X=a)=0\). Two consequences follow:
The value \(f(x)\) is not a probability. It is probability per unit length near \(x\). For a short interval of length \(\varepsilon\) centered at \(x\), where \(f\) is roughly constant,
\[P\left(x-\tfrac{\varepsilon}{2}<X<x+\tfrac{\varepsilon}{2}\right)=\int_{x-\varepsilon/2}^{x+\varepsilon/2}f(t)\,dt\approx f(x)\,\varepsilon.\]If \(X\) is measured in centimeters, \(f(x)\) has units of \(\text{cm}^{-1}\); multiplying by the length \(dx\) in an integral returns a dimensionless probability. A density can exceed 1 as long as the region where it is large is short. For example, the \(\operatorname{Unif}(0,1/2)\) density equals 2 on its support.
Incorrect: \(f(x)\) is the probability that \(X=x\), so it must lie between 0 and 1.
Correction: For a continuous random variable, \(P(X=x)=0\). The density \(f(x)\) is a rate: probability per unit length. It can be larger than 1. Only areas under \(f\) are probabilities.
Because \(f=F'\) and \(F(-\infty)=0\), the fundamental theorem of calculus gives
\[F(x)=\int_{-\infty}^{x}f(t)\,dt.\]The CDF is the area under the PDF to the left of \(x\). The variable \(t\) is a dummy variable, renamed to avoid a clash with the upper limit \(x\). For an interval,
The same probability appears as an area under the PDF and as a rise in the CDF.
Let \(f(x)=cx^2\) for \(0<x<2\) and \(f(x)=0\) otherwise. Find \(c\), the CDF, and \(P(0.5<X<1.5)\).
Step 1: Normalize. Nonnegativity holds for \(c>0\). The total area must be 1:
\[\int_0^2cx^2\,dx=c\frac{x^3}{3}\Big|_0^2=\frac{8c}{3}=1\implies c=\frac38.\]Step 2: CDF. For \(0<x<2\),
\[F(x)=\int_0^x\frac38t^2\,dt=\frac{x^3}{8},\]with \(F(x)=0\) for \(x\le0\) and \(F(x)=1\) for \(x\ge2\). Check: \(F'(x)=3x^2/8=f(x)\).
Step 3: Probability.
\[P(0.5<X<1.5)=F(1.5)-F(0.5)=\frac{3.375-0.125}{8}=\frac{13}{32}\approx0.406.\]Observation. The density at \(x=2\) is \(f(2)=3/2>1\), which is allowed.
For a continuous random variable \(X\) with PDF \(f\),
\[E(X)=\int_{-\infty}^{\infty}xf(x)\,dx,\]provided the integral converges absolutely. In practice, integrate over the support only.
The integrand weights each value \(x\) by the probability \(f(x)\,dx\) of a short interval near it, exactly as \(xP(X=x)\) does in the discrete case. The mean is the balancing point of the PDF. Linearity of expectation holds for continuous random variables as well.
For the density \(f(x)=3x^2/8\) on \((0,2)\):
\[E(X)=\int_0^2x\cdot\frac38x^2\,dx=\frac38\cdot\frac{x^4}{4}\Big|_0^2=\frac38\cdot4=\frac32.\]The mean lies to the right of the midpoint 1, because the density puts more weight near 2.
The mean locates a distribution but says nothing about its spread. Two random variables can share a mean and differ greatly in how far they typically fall from it.
Let \(\mu=E(X)\). The variance of \(X\) is
\[\operatorname{Var}(X)=E\left((X-\mu)^2\right),\]and the standard deviation is \(\operatorname{SD}(X)=\sqrt{\operatorname{Var}(X)}\).
Each part of the definition has a reason:
Proof. Expand the square and use linearity, treating \(\mu\) as a constant:
\[\begin{aligned} \operatorname{Var}(X) &=E\left(X^2-2\mu X+\mu^2\right)\\ &=E(X^2)-2\mu E(X)+\mu^2\\ &=E(X^2)-2\mu^2+\mu^2=E(X^2)-\mu^2. \end{aligned}\] \[\blacksquare\]Since \(\operatorname{Var}(X)\ge0\), the formula also shows \(E(X^2)\ge(EX)^2\). The notation \(EX^2\) means \(E(X^2)\): square first, then average.
For constants \(a\) and \(c\):
| Property | Reason |
|---|---|
| \(\operatorname{Var}(X+c)=\operatorname{Var}(X)\) | Shifting moves the mean by the same amount, leaving every deviation unchanged |
| \(\operatorname{Var}(aX)=a^2\operatorname{Var}(X)\) | Every deviation is multiplied by \(a\), so squared deviations are multiplied by \(a^2\) |
| \(\operatorname{Var}(X)\ge0\), with equality only if \(X\) is constant with probability 1 | Average of a nonnegative quantity |
| \(\operatorname{Var}(X+Y)=\operatorname{Var}(X)+\operatorname{Var}(Y)\) if \(X,Y\) independent | Proved later; fails in general without independence |
Variance is not linear. For example, \(\operatorname{Var}(X+X)=\operatorname{Var}(2X)=4\operatorname{Var}(X)\), not \(2\operatorname{Var}(X)\), unless \(\operatorname{Var}(X)=0\).
Fair die. Let \(X\) be uniform on \(\{1,\ldots,6\}\). Then \(E(X)=3.5\) and
\[E(X^2)=\frac{1^2+2^2+\cdots+6^2}{6}=\frac{91}{6}.\]So \(\operatorname{Var}(X)=\frac{91}{6}-\frac{49}{4}=\frac{182-147}{12}=\frac{35}{12}\approx2.917\) and \(\operatorname{SD}(X)\approx1.708\).
Continuous density. For \(f(x)=3x^2/8\) on \((0,2)\), with \(E(X)=3/2\),
\[E(X^2)=\int_0^2x^2\cdot\frac38x^2\,dx=\frac38\cdot\frac{32}{5}=\frac{12}{5}, \qquad \operatorname{Var}(X)=\frac{12}{5}-\frac94=\frac{3}{20}.\]So \(\operatorname{SD}(X)=\sqrt{0.15}\approx0.387\).
Incorrect: \(E(X^2)=(EX)^2\), so the variance formula is always 0.
Correction: Squaring is nonlinear. \(E(X^2)\) averages the squares, while \((EX)^2\) squares the average. Their difference is exactly the variance, which is positive unless \(X\) is constant.
Computing \(E(X^2)\) raises a question. \(X^2\) is a new random variable, so the definition of expectation asks for the distribution of \(X^2\). LOTUS avoids that step.
For a function \(g:\mathbb R\to\mathbb R\),
\[E(g(X))=\sum_xg(x)P(X=x)\quad\text{(discrete)},\qquad E(g(X))=\int_{-\infty}^{\infty}g(x)f(x)\,dx\quad\text{(continuous)}.\]The weights stay those of \(X\); only the values change from \(x\) to \(g(x)\).
Why it holds in the discrete case. Group the values \(x\) by the value \(y=g(x)\) they produce. Then \(P(g(X)=y)=\sum_{x:\,g(x)=y}P(X=x)\), so
\[\sum_yyP(g(X)=y)=\sum_y\sum_{x:\,g(x)=y}g(x)P(X=x)=\sum_xg(x)P(X=x).\]Grouping or not grouping gives the same total. The continuous version replaces sums with integrals; its proof is deferred in the textbook.
The name refers to how easily it is used without thinking: it looks like substituting \(g(x)\) for \(x\) in the definition, which is correct, but it is a theorem, not a definition.
Incorrect: \(E(g(X))=g(E(X))\), so LOTUS is unnecessary.
Correction: This equality holds for linear \(g\) but generally fails otherwise. For example, \(E(X^2)\ne(EX)^2\) unless the variance is 0. LOTUS applies \(g\) to each value before averaging.
A continuous random variable \(U\) is Uniform on \((a,b)\), written \(U\sim\operatorname{Unif}(a,b)\), if its PDF is constant on \((a,b)\) and 0 elsewhere:
\[f(x)=\begin{cases}\dfrac1{b-a}, & a<x<b,\\[4pt] 0, & \text{otherwise.}\end{cases}\]Why the constant is \(1/(b-a)\). Constant density \(c\) on an interval of length \(b-a\) gives total area \(c(b-a)\), which must equal 1.
CDF. Integrating the constant density gives a linear ramp:
\[F(x)=\begin{cases}0, & x\le a,\\[2pt] \dfrac{x-a}{b-a}, & a<x<b,\\[6pt] 1, & x\ge b.\end{cases}\]The standard Uniform \(\operatorname{Unif}(0,1)\) has \(f(x)=1\) and \(F(x)=x\) on \((0,1)\).
For a subinterval \((c,d)\subseteq(a,b)\),
\[P(c<U<d)=\frac{d-c}{b-a}.\]Only the length of the subinterval matters, not its location. This is what “completely random point” means, and among continuous distributions it characterizes the Uniform.
Let \(U\sim\operatorname{Unif}(0,1)\). Using the definition and LOTUS with \(f(x)=1\) on \((0,1)\):
\[E(U)=\int_0^1x\,dx=\frac12,\qquad E(U^2)=\int_0^1x^2\,dx=\frac13,\]so
Direct integration gives
\[E(U)=\int_a^b\frac{x}{b-a}\,dx=\frac{b^2-a^2}{2(b-a)}=\frac{a+b}{2},\]the midpoint of the interval, as symmetry requires. By LOTUS,
\[E(U^2)=\int_a^b\frac{x^2}{b-a}\,dx=\frac{b^3-a^3}{3(b-a)}=\frac{a^2+ab+b^2}{3}.\]Therefore
\[\operatorname{Var}(U)=\frac{a^2+ab+b^2}{3}-\frac{(a+b)^2}{4} =\frac{4a^2+4ab+4b^2-3a^2-6ab-3b^2}{12} =\frac{(b-a)^2}{12}.\]The variance depends only on the length of the interval, consistent with the fact that shifting does not change spread.
Textbook Section 5.2 obtains the general result from the standard case. If \(U\sim\operatorname{Unif}(0,1)\), then \(a+(b-a)U\sim\operatorname{Unif}(a,b)\): scaling by \(b-a\) stretches the interval to length \(b-a\) and adding \(a\) shifts it. Then
\[E(a+(b-a)U)=a+\frac{b-a}2=\frac{a+b}2,\qquad \operatorname{Var}(a+(b-a)U)=(b-a)^2\cdot\frac1{12}.\]This avoids the algebra above and uses only the variance properties in Part 5. The trick does not work for nonlinear transformations: \(U^2\) is not Uniform on \((0,1)\).
A shuttle departs at a time Uniform between 2 and 6 minutes after you reach the stop, so the wait is \(W\sim\operatorname{Unif}(2,6)\).
Interval probability. The interval \((3,4.5)\) has length 1.5 out of 4:
\[P(3<W<4.5)=\frac{1.5}{4}=0.375.\]Mean and spread.
\[E(W)=\frac{2+6}{2}=4,\qquad\operatorname{Var}(W)=\frac{4^2}{12}=\frac43,\qquad\operatorname{SD}(W)=\frac{2}{\sqrt3}\approx1.155.\]A LOTUS calculation. The expected squared wait is
\[E(W^2)=\operatorname{Var}(W)+(EW)^2=\frac43+16=\frac{52}{3}\approx17.33,\]which differs from \((EW)^2=16\).
Incorrect: Since every point of \((a,b)\) is equally likely, \(P(U=x)=1/(b-a)\).
Correction: \(1/(b-a)\) is the density, not a probability, and \(P(U=x)=0\). “Equally likely” for a continuous Uniform means that intervals of equal length have equal probability.
Try these before reading the answers.
Normalize: \(\int_{-1}^1(1-x^2)\,dx=2-\frac23=\frac43\), so \(c=\frac34\). Then
\[P(X>\tfrac12)=\frac34\int_{1/2}^1(1-x^2)\,dx=\frac34\left[\left(1-\frac13\right)-\left(\frac12-\frac1{24}\right)\right]=\frac34\cdot\frac5{24}=\frac5{32}.\]The density is symmetric about 0, so \(E(X)=0\). By LOTUS,
\[E(X^2)=\frac34\int_{-1}^1(x^2-x^4)\,dx=\frac34\left(\frac23-\frac25\right)=\frac15,\]so \(\operatorname{Var}(X)=\frac15\).
Yes. It is nonnegative and its area is \(2\times\frac12=1\). It is the \(\operatorname{Unif}(0,1/2)\) PDF. The density value 2 is not a probability.
Differentiate: \(f(x)=2x^{-3}\) for \(x>1\). Then
\[E(X)=\int_1^\infty x\cdot2x^{-3}\,dx=\int_1^\infty2x^{-2}\,dx=2, \qquad E(X^2)=\int_1^\infty2x^{-1}\,dx=\infty.\]The mean is finite but the second moment diverges, so the variance is infinite. A finite mean does not guarantee a finite variance.
\(\operatorname{Var}(3X+2)=9\cdot4=36\) and \(\operatorname{SD}(3X+2)=6\). The shift by 2 has no effect; the scale factor 3 is squared in the variance.
\(X^2=X\) because \(X\in\{0,1\}\), so \(E(X^2)=p\) and \(\operatorname{Var}(X)=p-p^2=p(1-p)\). This is maximized at \(p=1/2\), where it equals \(1/4\).
Here \(\sqrt{E(U)}=\sqrt{1/2}\approx0.707>2/3\). Applying a nonlinear function before or after averaging gives different results.
Given \(U<0.6\), the variable is \(\operatorname{Unif}(0,0.6)\), and probability is still proportional to length.
Every outcome of \(U\) is some specific number, and each specific number has probability 0. An event of probability 0 can still occur. Impossible events are those outside the sample space or support; \(0.5\) is inside \((0,1)\).
\(E(X+c)=\mu+c\), so \((X+c)-E(X+c)=X-\mu\). Hence \(\operatorname{Var}(X+c)=E\left((X-\mu)^2\right)=\operatorname{Var}(X)\).
| Concept | Essential fact |
|---|---|
| Continuous random variable | Differentiable CDF; \(P(X=x)=0\) for every \(x\) |
| \(f=F'\); probability per unit length | |
| Valid PDF | \(f\ge0\) and \(\int f=1\) |
| PDF to CDF | \(F(x)=\int_{-\infty}^xf(t)\,dt\) |
| Interval probability | \(P(a<X\le b)=F(b)-F(a)=\int_a^bf\) |
| Endpoints | Irrelevant for continuous, relevant for discrete |
| Density above 1 | Allowed; only areas are probabilities |
| Continuous mean | \(E(X)=\int xf(x)\,dx\) |
| Variance | \(E((X-\mu)^2)=E(X^2)-(EX)^2\) |
| Standard deviation | \(\sqrt{\operatorname{Var}(X)}\), in the units of \(X\) |
| Shift and scale | \(\operatorname{Var}(aX+c)=a^2\operatorname{Var}(X)\) |
| LOTUS | \(E(g(X))=\int g(x)f(x)\,dx\) or \(\sum g(x)P(X=x)\) |
| Nonlinear \(g\) | Generally \(E(g(X))\ne g(EX)\) |
| Uniform PDF | \(1/(b-a)\) on \((a,b)\) |
| Uniform CDF | \((x-a)/(b-a)\) on \((a,b)\) |
| Uniform probability | Length of interval divided by \(b-a\) |
| Uniform mean and variance | \((a+b)/2\) and \((b-a)^2/12\) |
| Standard Uniform | Mean \(1/2\), variance \(1/12\) |
A random variable whose CDF is differentiable, except possibly at finitely many points where it is still continuous. It assigns probability 0 to each single value.
The derivative \(f=F'\) of the CDF of a continuous random variable. Integrating it over a region gives the probability of that region.
The set of values where the PMF or PDF is positive.
The expected squared deviation from the mean, \(E((X-\mu)^2)\), equivalently \(E(X^2)-(EX)^2\).
The square root of the variance, measuring spread in the units of the random variable.
The expectation of \(X^2\), also called the second moment. It is not \((EX)^2\).
The law of the unconscious statistician: \(E(g(X))\) is computed by weighting \(g(x)\) with the PMF or PDF of \(X\), without finding the distribution of \(g(X)\).
The distribution of a completely random point in an interval \((a,b)\), with constant density \(1/(b-a)\), so that probability is proportional to length.
A transformation \(Y=\sigma X+\mu\) with \(\sigma>0\), which shifts and rescales a distribution. It preserves the Uniform family.
A single \(\operatorname{Unif}(0,1)\) random variable can be turned into a random variable with any continuous distribution by applying the inverse CDF. In the other direction, plugging a continuous random variable into its own CDF produces a \(\operatorname{Unif}(0,1)\) random variable. Independence of random variables means that the joint CDF factors into marginal CDFs, which is stronger than pairwise independence. The standard Normal distribution has density proportional to \(e^{-z^2/2}\). A polar-coordinates argument finds its normalizing constant \(1/\sqrt{2\pi}\), odd symmetry gives mean 0, and integration by parts gives variance 1.
| Symbol | Meaning |
|---|---|
| \(U\sim\operatorname{Unif}(0,1)\) | Standard Uniform random variable |
| \(F\) | A CDF, continuous and strictly increasing on its support |
| \(F^{-1}\) | Inverse CDF, also called the quantile function |
| \(F(X)\) | The random variable obtained by plugging \(X\) into the function \(F\) |
| \(X_1,\ldots,X_n\) | Random variables defined on the same experiment |
| i.i.d. | Independent and identically distributed |
| \(Z\sim\mathcal N(0,1)\) | \(Z\) has the standard Normal distribution |
| \(\varphi(z)\) | Standard Normal PDF, \(\frac1{\sqrt{2\pi}}e^{-z^2/2}\) |
| \(\Phi(z)\) | Standard Normal CDF, \(\int_{-\infty}^z\varphi(t)\,dt\) |
| \((r,\theta)\) | Polar coordinates, with \(x=r\cos\theta\), \(y=r\sin\theta\) |
After studying these notes, you should be able to:
Let \(F\) be a CDF that is continuous and strictly increasing on the support of its distribution, so that \(F^{-1}:(0,1)\to\mathbb R\) exists.
The theorem says that \(\operatorname{Unif}(0,1)\) is a universal raw material: every continuous distribution can be built from it, and every continuous random variable can be converted back to it. It is also called the probability integral transform or inverse transform sampling.
Proof of part 1. Because \(F\) is strictly increasing, \(F^{-1}(U)\le x\) holds exactly when \(U\le F(x)\). For a standard Uniform, \(P(U\le u)=u\) for \(u\in(0,1)\). Hence
\[P(X\le x)=P\left(F^{-1}(U)\le x\right)=P\left(U\le F(x)\right)=F(x).\]Proof of part 2. Let \(Y=F(X)\), which takes values in \((0,1)\). For \(y\in(0,1)\), apply the increasing function \(F^{-1}\) to both sides of the inequality:
\[P(Y\le y)=P\left(F(X)\le y\right)=P\left(X\le F^{-1}(y)\right)=F\left(F^{-1}(y)\right)=y.\]This is the \(\operatorname{Unif}(0,1)\) CDF. \(\blacksquare\)
Both proofs use only one fact: an increasing function preserves inequalities, so an event about \(X\) can be rewritten as an event about \(U\).
\(F\) is an ordinary function, so \(F(X)\) is a function of a random variable and hence a random variable. First write \(F(x)\) as a formula in \(x\), then substitute \(X\). For example, if \(F(x)=1-e^{-x}\) for \(x>0\), then \(F(X)=1-e^{-X}\).
Incorrect: \(F(X)=P(X\le X)=1\).
Correction: \(F(x)=P(X\le x)\) defines a function of a fixed number \(x\). Replacing \(x\) inside the probability by the random \(X\) confuses the function with the event that defines it. The correct object is the formula \(F(x)\) evaluated at \(x=X\), which is a random number between 0 and 1.
Interpretation as percentiles. If \(X\) is the score of a randomly chosen student and \(F\) is the score distribution, then \(F(x)\) is the fraction of students scoring below \(x\), the percentile of score \(x\). \(F(X)\) is the percentile of a random student. Scores can have any shape, but percentiles are spread evenly: 10% of students lie in each tenth of the percentile scale. That is part 2 of the theorem.
The \(\operatorname{Expo}(1)\) distribution has CDF \(F(x)=1-e^{-x}\) for \(x>0\). Generate draws from it using only Uniform draws.
Step 1: Invert the CDF. Solve \(u=1-e^{-x}\) for \(x\):
\[e^{-x}=1-u\implies x=-\ln(1-u),\qquad F^{-1}(u)=-\ln(1-u).\]Step 2: Substitute the Uniform. By part 1 of the theorem,
\[X=-\ln(1-U)\sim\operatorname{Expo}(1).\]Step 3: Check directly. For \(x>0\),
\[P(-\ln(1-U)\le x)=P(1-U\ge e^{-x})=P(U\le1-e^{-x})=1-e^{-x}.\]Step 4: Simplify (optional). Since \(1-U\sim\operatorname{Unif}(0,1)\) as well, \(-\ln U\sim\operatorname{Expo}(1)\) too. To simulate ten independent draws, apply the transformation to ten independent Uniforms.
Reverse direction. If \(X\sim\operatorname{Expo}(1)\), part 2 gives \(1-e^{-X}\sim\operatorname{Unif}(0,1)\).
Large values of \(u\) near 1 map to widely spaced \(x\)-values because the CDF is flat in the right tail. A flat region of \(F\) means little probability per unit length, so the inverse spreads Uniform mass thinly there.
A discrete CDF has jumps and flat parts, so it has no ordinary inverse. Part 1 still works with a different construction. To build \(X\) with \(P(X=j)=p_j\) for \(j=0,1,\ldots\), cut \((0,1)\) into consecutive pieces of lengths \(p_0,p_1,\ldots\) and let \(X=j\) when \(U\) falls in the \(j\)th piece. Since probability is length for \(U\), \(P(X=j)=p_j\). For example, \(X=I(U<p)\) is \(\operatorname{Bern}(p)\).
Part 2 fails for discrete \(X\): \(F(X)\) is still discrete. If \(X\sim\operatorname{Bern}(p)\), then \(F(X)\) takes only the values \(1-p\) and \(1\).
Random variables \(X_1,\ldots,X_n\) are independent if
\[P(X_1\le x_1,\ldots,X_n\le x_n)=P(X_1\le x_1)\cdots P(X_n\le x_n)\quad\text{for all }x_1,\ldots,x_n\in\mathbb R.\]For discrete random variables, this is equivalent to factorization of the joint PMF:
\[P(X_1=x_1,\ldots,X_n=x_n)=P(X_1=x_1)\cdots P(X_n=x_n)\quad\text{for all }x_1,\ldots,x_n.\]Infinitely many random variables are independent if every finite subcollection is.
Meaning. Learning the values of any subset of the variables does not change the distribution of the others.
One equation, infinitely many conditions. For \(n\) events, independence requires checking every subcollection of 2, 3, …, \(n\) events. For random variables, one product formula suffices because it must hold for every choice of \(x_1,\ldots,x_n\). Letting some \(x_k\to\infty\) removes those variables from the statement, so the definition already includes every subcollection. A single choice of values where the formula fails proves dependence.
i.i.d. Random variables are independent and identically distributed when they are independent and share one distribution. The two properties are unrelated: independence concerns information, identical distribution concerns the shape of the PMF or CDF.
Roll two fair dice. Let \(M\) be the larger value and \(L\) the smaller. Are \(M\) and \(L\) independent?
Find one pair of values where the joint PMF does not factor. Take \(M=1\) and \(L=6\). The maximum cannot be below the minimum, so \(P(M=1,L=6)=0\). But \(P(M=1)=1/36\) (both dice show 1) and \(P(L=6)=1/36\) (both dice show 6), so
\[P(M=1)P(L=6)=\frac1{1296}\ne0.\]Hence \(M\) and \(L\) are dependent.
Let \(X_1,X_2\) be i.i.d. \(\operatorname{Bern}(1/2)\), two fair coin tosses coded 0 or 1. Let \(X_3=1\) if \(X_1=X_2\) and \(X_3=0\) otherwise.
Each variable is \(\operatorname{Bern}(1/2)\). \(X_3=1\) for the outcomes \((0,0)\) and \((1,1)\), so \(P(X_3=1)=1/2\).
Each pair is independent. \(X_1\) and \(X_2\) are independent by construction. For \(X_1\) and \(X_3\), the four joint probabilities are each \(1/4\), since each joint value pins down one equally likely outcome. For instance,
\[P(X_1=1,X_3=1)=P(X_1=1,X_2=1)=\frac14=P(X_1=1)P(X_3=1).\]The same argument applies to \(X_2\) and \(X_3\).
The three are not independent. Knowing \(X_1\) and \(X_2\) determines \(X_3\). For instance,
\[P(X_1=1,X_2=1,X_3=0)=0\ne\frac18=P(X_1=1)P(X_2=1)P(X_3=0).\]Incorrect: If every pair among \(X_1,\ldots,X_n\) is independent, then the collection is independent.
Correction: Pairwise independence does not imply joint independence, as matching pennies shows. The converse does hold: independence of the collection implies independence of every pair and every subcollection.
The Normal distribution, also called the Gaussian distribution, is the most important distribution in statistics. Its importance comes from the central limit theorem: under mild conditions, a sum of many i.i.d. random variables, suitably standardized, has an approximately Normal distribution, whatever the distribution of each term, discrete or continuous. Sums and averages appear throughout statistics, so the Normal appears with them.
The Normal family has two parameters, the mean \(\mu\) and the variance \(\sigma^2\), written \(\mathcal N(\mu,\sigma^2)\). Its PDF is a symmetric bell centered at \(\mu\), with spread controlled by \(\sigma\). The standard case \(\mu=0\), \(\sigma^2=1\) is treated first.
Start from the shape \(e^{-z^2/2}\): positive, symmetric about 0, and decaying very fast. To make it a PDF, multiply by a constant \(c\) chosen so the total area is 1:
\[f(z)=ce^{-z^2/2},\qquad c=\left(\int_{-\infty}^{\infty}e^{-z^2/2}\,dz\right)^{-1}.\]The antiderivative of \(e^{-z^2/2}\) has no closed form, but the definite integral over the whole line can still be computed.
Let \(I=\int_{-\infty}^{\infty}e^{-z^2/2}\,dz\). The integration variable is a dummy variable, so write \(I\) twice with different names and multiply:
\[I^2=\int_{-\infty}^{\infty}e^{-x^2/2}\,dx\int_{-\infty}^{\infty}e^{-y^2/2}\,dy=\int_{-\infty}^{\infty}\int_{-\infty}^{\infty}e^{-(x^2+y^2)/2}\,dx\,dy.\]The integrand depends only on the distance \(r=\sqrt{x^2+y^2}\) from the origin, so switch to polar coordinates. The area element becomes \(dx\,dy=r\,dr\,d\theta\), where the factor \(r\) is the Jacobian:
\[I^2=\int_0^{2\pi}\int_0^{\infty}e^{-r^2/2}\,r\,dr\,d\theta.\]The extra \(r\) makes the inner integral elementary. Substitute \(u=r^2/2\), so \(du=r\,dr\):
\[\int_0^{\infty}e^{-r^2/2}\,r\,dr=\int_0^{\infty}e^{-u}\,du=1.\]The outer integral then gives \(I^2=\int_0^{2\pi}1\,d\theta=2\pi\). Since \(I>0\), \(I=\sqrt{2\pi}\).
\(Z\) has the standard Normal distribution, \(Z\sim\mathcal N(0,1)\), if its PDF is
\[\varphi(z)=\frac1{\sqrt{2\pi}}e^{-z^2/2},\qquad-\infty<z<\infty.\]Its CDF is
\[\Phi(z)=\int_{-\infty}^{z}\frac1{\sqrt{2\pi}}e^{-t^2/2}\,dt.\]\(\Phi\) has no closed form in elementary functions, but it is a well-defined function, tabulated and built into software. Reference values: \(\Phi(0)=0.5\), \(\Phi(1)\approx0.8413\), \(\Phi(1.96)\approx0.9750\), \(\Phi(2)\approx0.9772\).
Because \(\varphi(-z)=\varphi(z)\), the area to the left of \(-z\) equals the area to the right of \(z\):
So \(P(\lvert Z\rvert<z)=\Phi(z)-\Phi(-z)=2\Phi(z)-1\). For example, \(P(\lvert Z\rvert<1)\approx0.6827\) and \(P(\lvert Z\rvert<1.96)\approx0.95\). (Supplementary: the same symmetry shows that \(-Z\sim\mathcal N(0,1)\).)
The integrand \(g(z)=ze^{-z^2/2}\) is odd: \(g(-z)=-g(z)\). Its area on \((-\infty,0)\) is the negative of its area on \((0,\infty)\), and each of these areas is finite: \(\int_0^\infty ze^{-z^2/2}\,dz=1\) by the substitution \(u=z^2/2\). Hence
The finiteness check matters, since \(\infty-\infty\) is undefined. The same reasoning gives \(E(Z^n)=0\) for every odd \(n\).
Since \(E(Z)=0\), LOTUS gives
\[\operatorname{Var}(Z)=E(Z^2)=\frac1{\sqrt{2\pi}}\int_{-\infty}^{\infty}z^2e^{-z^2/2}\,dz=\frac2{\sqrt{2\pi}}\int_0^{\infty}z^2e^{-z^2/2}\,dz,\]where the last step uses that \(z^2e^{-z^2/2}\) is even.
Integrate by parts, splitting \(z^2e^{-z^2/2}=z\cdot\left(ze^{-z^2/2}\right)\):
| Choice | Derivative or antiderivative |
|---|---|
| \(u=z\) | \(du=dz\) |
| \(dv=ze^{-z^2/2}\,dz\) | \(v=-e^{-z^2/2}\) |
Then
\[\int_0^{\infty}z^2e^{-z^2/2}\,dz=\Big[-ze^{-z^2/2}\Big]_0^{\infty}+\int_0^{\infty}e^{-z^2/2}\,dz=0+\frac{\sqrt{2\pi}}2.\]The boundary term vanishes because \(e^{-z^2/2}\) decays faster than \(z\) grows. The remaining integral is half of \(I=\sqrt{2\pi}\) by symmetry. Therefore
The notation \(\mathcal N(0,1)\) records exactly these two facts: mean 0 and variance 1.
Incorrect: Any integral of an odd function over \((-\infty,\infty)\) equals 0.
Correction: The two half-line integrals must be finite. For example, \(\int_{-\infty}^{\infty}z\,dz\) is undefined, not 0. For the Normal, the rapid decay of \(e^{-z^2/2}\) makes every moment finite, so the symmetry argument is valid.
Incorrect: Since \(\varphi(0)=1/\sqrt{2\pi}\approx0.399\), the probability that \(Z=0\) is about 0.4.
Correction: \(\varphi(0)\) is a density. \(P(Z=0)=0\), and \(P(-\varepsilon/2<Z<\varepsilon/2)\approx0.399\,\varepsilon\) for small \(\varepsilon\).
Try these before reading the answers.
Solve \(u=x^3\): \(F^{-1}(u)=u^{1/3}\). So \(X=U^{1/3}\). Check: for \(0<x<1\), \(P(U^{1/3}\le x)=P(U\le x^3)=x^3\).
Solve \(u=1-1/x\): \(x=1/(1-u)\). So \(X=1/(1-U)\) has the required CDF, and so does \(1/U\) by symmetry of the Uniform.
By part 2 of the theorem, \(F(X)=1-e^{-X}\sim\operatorname{Unif}(0,1)\). Then \(e^{-X}=1-F(X)\) is also \(\operatorname{Unif}(0,1)\), because \(1-U\) is Uniform when \(U\) is.
Let \(X=1\) if \(U<0.2\), \(X=2\) if \(0.2\le U<0.7\), and \(X=3\) if \(U\ge0.7\). The three pieces have lengths \(0.2,0.5,0.3\), which are the required probabilities.
They are dependent. \(P(X=1,X+Y=12)=0\), since a sum of 12 requires both dice to show 6. But \(P(X=1)P(X+Y=12)=\frac16\cdot\frac1{36}>0\).
No. The pair \((X_2,X_3)\) determines \(X_1\): \(X_1=X_2\) if \(X_3=1\) and \(X_1=1-X_2\) if \(X_3=0\). For example, \(P(X_1=0\mid X_2=1,X_3=1)=0\ne\frac12\).
Substitute \(z=2w\), \(dz=2\,dw\): \(\int e^{-z^2/8}\,dz=2\int e^{-w^2/2}\,dw=2\sqrt{2\pi}\). So \(c=\frac1{2\sqrt{2\pi}}\). This is the \(\mathcal N(0,4)\) density.
\(P(-0.5<Z<1.5)=\Phi(1.5)-\Phi(-0.5)=\Phi(1.5)+\Phi(0.5)-1\approx0.9332+0.6915-1=0.6247\).
Write \(E(Z^4)=\frac1{\sqrt{2\pi}}\int z^3\cdot ze^{-z^2/2}\,dz\) and integrate by parts with \(u=z^3\) and \(v=-e^{-z^2/2}\). The boundary term vanishes, leaving
\[E(Z^4)=\frac1{\sqrt{2\pi}}\int_{-\infty}^{\infty}3z^2e^{-z^2/2}\,dz=3E(Z^2)=3.\]| Concept | Essential fact |
|---|---|
| Universality, part 1 | \(U\sim\operatorname{Unif}(0,1)\implies F^{-1}(U)\sim F\) |
| Universality, part 2 | \(X\sim F\) continuous \(\implies F(X)\sim\operatorname{Unif}(0,1)\) |
| Key step in proofs | Increasing functions preserve inequalities |
| \(F(X)\) | Formula \(F(x)\) evaluated at \(x=X\); not \(P(X\le X)\) |
| Exponential simulation | \(-\ln(1-U)\) or \(-\ln U\sim\operatorname{Expo}(1)\) |
| Uniform symmetry | \(1-U\sim\operatorname{Unif}(0,1)\) |
| Discrete simulation | Split \((0,1)\) into pieces of lengths \(p_j\) |
| Independence | Joint CDF (or PMF) factors for all values |
| Dependence | One failing pair of values suffices |
| Pairwise vs. joint | Pairwise independence does not imply independence |
| Normal importance | Central limit theorem for sums of i.i.d. variables |
| Standard Normal PDF | \(\varphi(z)=e^{-z^2/2}/\sqrt{2\pi}\) |
| Gaussian integral | \(\int e^{-z^2/2}\,dz=\sqrt{2\pi}\) via polar coordinates |
| Symmetry | \(\Phi(-z)=1-\Phi(z)\) |
| Moments | \(E(Z)=0\), \(E(Z^2)=\operatorname{Var}(Z)=1\) |
| Reference values | \(\Phi(1)\approx0.841\), \(\Phi(1.96)\approx0.975\) |
The pair of results \(F^{-1}(U)\sim F\) for \(U\sim\operatorname{Unif}(0,1)\), and \(F(X)\sim\operatorname{Unif}(0,1)\) for continuous \(X\) with CDF \(F\).
The inverse CDF \(F^{-1}\), which maps a probability level \(u\) to the value with that fraction of the distribution below it.
Generating draws from a distribution by applying its quantile function to Uniform draws.
Factorization of the joint CDF into marginal CDFs for all arguments; for discrete variables, equivalently, factorization of the joint PMF.
Independence of every pair in a collection. It is weaker than independence of the whole collection.
Independent and identically distributed: independent random variables sharing a single distribution.
The distribution with PDF \(\varphi(z)=e^{-z^2/2}/\sqrt{2\pi}\), mean 0, and variance 1, written \(\mathcal N(0,1)\).
The standard Normal PDF and CDF. \(\Phi\) has no elementary closed form.
The constant multiplying a nonnegative function so that it integrates (or sums) to 1. For the standard Normal it is \(1/\sqrt{2\pi}\).
The result that a standardized sum of many i.i.d. random variables is approximately standard Normal. It explains the central role of the Normal distribution.
Every Normal distribution is a shifted and rescaled standard Normal: \(X=\mu+\sigma Z\). Location-scale reasoning gives its mean, variance, CDF, and PDF directly from the standard case, and standardization turns any Normal probability into a statement about \(\Phi\). The 68-95-99.7% rule summarizes how much probability lies within one, two, and three standard deviations of the mean. LOTUS computes \(E(g(X))\) by keeping the probabilities of \(X\) and changing only the values. Combined with series manipulation and indicator decompositions, it gives the variances of the Poisson and Binomial distributions.
| Symbol | Meaning |
|---|---|
| \(Z\sim\mathcal N(0,1)\) | Standard Normal random variable |
| \(\varphi\), \(\Phi\) | Standard Normal PDF and CDF |
| \(X\sim\mathcal N(\mu,\sigma^2)\) | Normal with mean \(\mu\) and variance \(\sigma^2\) |
| \(\mu\) | Location parameter (mean) |
| \(\sigma>0\) | Scale parameter (standard deviation) |
| \((X-\mu)/\sigma\) | Standardized version of \(X\) |
| \(\Phi^{-1}\) | Standard Normal quantile function |
| \(g(X)\) | A function of a random variable |
| \(I_j\) | Indicator of success on trial \(j\) |
| \(q=1-p\) | Failure probability |
After studying these notes, you should be able to:
Variance measures squared distance from the mean, so a scale factor enters squared, while a shift has no effect:
A frequent error follows from treating variance as linear. Adding a random variable to itself is the same as doubling it:
\[\operatorname{Var}(X+X)=\operatorname{Var}(2X)=4\operatorname{Var}(X)\ne2\operatorname{Var}(X)\quad\text{unless }\operatorname{Var}(X)=0.\]The value \(2\operatorname{Var}(X)\) is what independent copies would give: if \(X_1,X_2\) are i.i.d. with the distribution of \(X\), then \(\operatorname{Var}(X_1+X_2)=2\operatorname{Var}(X)\). Two independent copies partly cancel each other’s deviations; \(X\) and itself never do.
Incorrect: \(X+X\) and \(X_1+X_2\) have the same variance because \(X\), \(X_1\), and \(X_2\) have the same distribution.
Correction: The variance of a sum depends on how the terms are related, not only on their individual distributions. \(X+X=2X\) has variance \(4\operatorname{Var}(X)\); an independent pair has variance \(2\operatorname{Var}(X)\). Treating the two as equal is sympathetic magic.
Let \(Z\sim\mathcal N(0,1)\), \(\mu\in\mathbb R\), and \(\sigma>0\). The random variable
\[X=\mu+\sigma Z\]has the Normal distribution with mean \(\mu\) and variance \(\sigma^2\), written \(X\sim\mathcal N(\mu,\sigma^2)\).
Mean and variance. Linearity and the scaling rule give
\[E(X)=\mu+\sigma E(Z)=\mu,\qquad\operatorname{Var}(X)=\sigma^2\operatorname{Var}(Z)=\sigma^2.\]The scale factor is the standard deviation \(\sigma\), not the variance \(\sigma^2\). Multiplying \(Z\) by \(\sigma^2\) would give variance \(\sigma^4\).
Parameter convention. The second parameter of \(\mathcal N(\mu,\sigma^2)\) is the variance. \(\mathcal N(0,4)\) has standard deviation 2. Software often takes the standard deviation instead, so check which one a function expects.
Standardization. Solving for \(Z\) reverses the construction:
The standardized value measures how many standard deviations \(X\) lies from its mean.
The CDF follows by standardizing inside the probability. Since \(\sigma>0\), dividing by \(\sigma\) preserves the inequality:
\[F(x)=P(X\le x)=P\left(\frac{X-\mu}{\sigma}\le\frac{x-\mu}{\sigma}\right)=\Phi\left(\frac{x-\mu}{\sigma}\right).\]Differentiate with the chain rule. The inner function \((x-\mu)/\sigma\) has derivative \(1/\sigma\):
The factor \(1/\sigma\) is essential. Stretching the curve horizontally by \(\sigma\) multiplies its area by \(\sigma\), so its height must be divided by \(\sigma\) to keep total area 1.
Incorrect: The PDF of \(\mu+\sigma Z\) is \(\mu+\sigma\varphi(z)\), obtained by shifting and scaling the density.
Correction: Location-scale operations act on the random variable, not on its PDF. Applying them to \(\varphi\) gives a function whose area is not 1. The correct density substitutes the standardized argument and divides by \(\sigma\): \(\frac1\sigma\varphi\left(\frac{x-\mu}\sigma\right)\).
Model adult heights as \(X\sim\mathcal N(170,64)\) in centimeters, so \(\mu=170\) and \(\sigma=8\).
(a) Probability above 186 cm. Standardize: \((186-170)/8=2\). Then
\[P(X>186)=P(Z>2)=1-\Phi(2)\approx1-0.9772=0.0228.\](b) Probability between 162 and 186 cm. The standardized endpoints are \(-1\) and \(2\):
\[P(162<X<186)=\Phi(2)-\Phi(-1)=\Phi(2)-(1-\Phi(1))\approx0.9772-0.1587=0.8186.\](c) The 90th percentile. Find \(x\) with \(P(X\le x)=0.9\). Since \(\Phi\left(\frac{x-170}8\right)=0.9\),
\[x=170+8\,\Phi^{-1}(0.9)\approx170+8(1.2816)\approx180.3\text{ cm}.\]For \(X\sim\mathcal N(\mu,\sigma^2)\), standardization gives \(P(\lvert X-\mu\rvert<k\sigma)=P(\lvert Z\rvert<k)=2\Phi(k)-1\). The values for \(k=1,2,3\) are worth memorizing:
More precisely, the three values are \(0.6827\), \(0.9545\), and \(0.9973\). The exact 95% multiplier is \(1.96\), since \(\Phi(1.96)\approx0.975\), which is why \(\pm1.96\) standard deviations appears in 95% confidence intervals.
The rule gives quick estimates. In the height example, \(162<X<186\) is \(\mu-\sigma\) to \(\mu+2\sigma\). Half the 68% band plus half the 95% band gives \(0.34+0.475=0.815\), close to the exact \(0.8186\).
The distribution of \(g(X)\) uses the same probabilities as \(X\), attached to transformed values. If \(X\) takes values \(0,1,2,3,\ldots\), then \(X^2\) takes \(0,1,4,9,\ldots\) with the same probabilities, so
\[E(X)=\sum_xxP(X=x),\qquad E(X^2)=\sum_xx^2P(X=x).\]This works without change when \(g\) is one-to-one on the support. When several values of \(x\) share one value of \(g(x)\), their probabilities must be combined to get the PMF of \(g(X)\). LOTUS states that the expectation comes out right either way.
If \(X\) is discrete and \(g:\mathbb R\to\mathbb R\), then
\[E(g(X))=\sum_xg(x)P(X=x),\]where the sum is over the support of \(X\).
Proof. Work with the sample space \(S\), finite or countable, each outcome \(s\) a “pebble” with mass \(P(\{s\})\). The random variable \(g(X)\) assigns the value \(g(X(s))\) to pebble \(s\), so
\[E(g(X))=\sum_{s\in S}g(X(s))P(\{s\}).\]Group the pebbles by their value of \(X\). All pebbles with \(X(s)=x\) form a “super-pebble” of total mass \(P(X=x)\), and every one of them has the same \(g\)-value \(g(x)\):
\[\sum_{s\in S}g(X(s))P(\{s\})=\sum_x\sum_{s:\,X(s)=x}g(x)P(\{s\})=\sum_xg(x)\sum_{s:\,X(s)=x}P(\{s\})=\sum_xg(x)P(X=x).\] \[\blacksquare\]The proof only reorders a sum. Any grouping of the pebbles gives the same total, so grouping by \(x\) (LOTUS) and grouping by \(g(x)\) (the definition of \(E(g(X))\)) must agree.
Let \(X\) be uniform on \(\{-2,-1,0,1,2\}\).
By LOTUS. Keep each probability \(1/5\) and square each value:
\[E(X^2)=\frac{4+1+0+1+4}{5}=2.\]By the distribution of \(X^2\). The values \(\pm1\) both map to 1, and \(\pm2\) both map to 4, so \(X^2\) has PMF \(P(X^2=0)=1/5\), \(P(X^2=1)=2/5\), \(P(X^2=4)=2/5\):
\[E(X^2)=0\cdot\frac15+1\cdot\frac25+4\cdot\frac25=2.\]Both routes give 2. Since \(E(X)=0\), this is also \(\operatorname{Var}(X)\).
Let \(X\sim\operatorname{Pois}(\lambda)\). Its mean is \(\lambda\) (Lecture 11). For the variance, LOTUS gives
\[E(X^2)=\sum_{k=0}^{\infty}k^2\frac{e^{-\lambda}\lambda^k}{k!}=e^{-\lambda}\sum_{k=0}^{\infty}k^2\frac{\lambda^k}{k!}.\]The sum \(\sum k^2\lambda^k/k!\) can be produced from the exponential series. Each application of the operation “differentiate with respect to \(\lambda\), then multiply by \(\lambda\)” brings down one factor of \(k\), because \(\lambda\frac{d}{d\lambda}\lambda^k=k\lambda^k\).
Step 1. Start from \(\sum_{k=0}^\infty\lambda^k/k!=e^\lambda\).
Step 2. Differentiate and multiply by \(\lambda\):
\[\sum_{k=0}^{\infty}k\frac{\lambda^k}{k!}=\lambda e^{\lambda}.\]Step 3. Repeat on both sides:
\[\sum_{k=0}^{\infty}k^2\frac{\lambda^k}{k!}=\lambda\frac{d}{d\lambda}\left(\lambda e^\lambda\right)=\lambda\left(e^\lambda+\lambda e^\lambda\right)=\lambda e^\lambda(1+\lambda).\]Term-by-term differentiation is valid because power series can be differentiated inside their interval of convergence, which here is the whole real line.
Step 4. Multiply by \(e^{-\lambda}\) and subtract the squared mean:
\[E(X^2)=\lambda(1+\lambda)=\lambda^2+\lambda,\qquad \operatorname{Var}(X)=\lambda^2+\lambda-\lambda^2=\lambda.\]Equal mean and variance is a characteristic feature of the Poisson. Lecture 11 reached the same result through \(E(X(X-1))\); the derivative method is a general tool for sums of the form \(\sum k^mc_k\lambda^k\).
Let \(X\sim\operatorname{Bin}(n,p)\) and write \(X=I_1+\cdots+I_n\), where \(I_j\) indicates success on trial \(j\). The \(I_j\) are i.i.d. \(\operatorname{Bern}(p)\), and \(E(X)=np\) by linearity.
Squaring the sum produces \(n\) squared terms and \(\binom n2\) distinct cross terms, each appearing twice:
\[X^2=\sum_{j=1}^nI_j^2+2\sum_{i<j}I_iI_j.\]Two facts make each expectation easy:
By linearity,
\[E(X^2)=np+2\binom n2p^2=np+n(n-1)p^2.\]Then
\[\operatorname{Var}(X)=np+n(n-1)p^2-n^2p^2=np-np^2=np(1-p).\]Each indicator has \(\operatorname{Var}(I_j)=E(I_j^2)-(EI_j)^2=p-p^2=pq\). For independent random variables variances add (proved later), so \(\operatorname{Var}(X)=npq\) immediately.
The variance is largest at \(p=1/2\) and vanishes when \(p=0\) or \(p=1\), when the outcome is certain. For \(\operatorname{Bin}(10,0.3)\), \(\operatorname{Var}(X)=10(0.3)(0.7)=2.1\).
Comparison with the Poisson. With \(np=\lambda\), the Binomial variance is \(\lambda(1-p)\), which tends to \(\lambda\) as \(p\to0\). This matches the Poisson limit of the Binomial from Lecture 11.
Incorrect: \(E(I_iI_j)=E(I_i)\) because indicators satisfy \(I^2=I\).
Correction: \(I^2=I\) holds for one indicator multiplied by itself. For \(i\ne j\), \(I_iI_j\) indicates that two different events both occur, with expectation \(P(A_i\cap A_j)\), which equals \(p^2\) only under independence.
For draws without replacement, the indicators are dependent, so \(E(I_iI_j)\ne p^2\) and the cross terms change. The same expansion still works with the correct joint probability; the Hypergeometric variance is derived later in the course.
Try these before reading the answers.
Here \(\sigma=3\). \(P(X>1)=P\left(Z>\frac{1-(-2)}3\right)=1-\Phi(1)\approx0.1587\). The second event is \(\lvert X-\mu\rvert<2\sigma\), so its probability is \(2\Phi(2)-1\approx0.9545\).
\(3-2X\sim\mathcal N(3-2\mu,4\sigma^2)\). The mean transforms linearly; the variance picks up \((-2)^2=4\).
\(P(X>x)=0.025\) means \(\Phi\left(\frac{x-50}5\right)=0.975\), so \(x\approx50+5(1.96)=59.8\).
\(X+Y\sim\mathcal N(3,9)\), standard deviation 3. \(P(X+Y>6)=1-\Phi(1)\approx0.1587\).
\(X-Y=X+(-1)Y\) and \(\operatorname{Var}(-Y)=(-1)^2\operatorname{Var}(Y)=\operatorname{Var}(Y)\). Uncertainty from both sources accumulates whether they are added or subtracted.
Note that this is not \(1/(\lambda+1)\): for \(\lambda=3\) it is \(0.317\), not \(0.25\).
The terms with \(k=0,1\) vanish, and \(k(k-1)/k!=1/(k-2)!\):
\[E(X(X-1))=e^{-\lambda}\sum_{k=2}^\infty\frac{\lambda^k}{(k-2)!}=\lambda^2e^{-\lambda}\sum_{j=0}^\infty\frac{\lambda^j}{j!}=\lambda^2.\]Then \(E(X^2)=\lambda^2+\lambda\), consistent with Part 5.
\(20(0.25)(0.75)=3.75\). The function \(p(1-p)\) is maximized at \(p=1/2\), giving variance \(5\).
Differentiate \(\sum_{k=0}^\infty q^k=\frac1{1-q}\): \(\sum_{k=1}^\infty kq^{k-1}=\frac1{(1-q)^2}=\frac1{p^2}\). So \(E(X)=p\cdot\frac1{p^2}=\frac1p\).
Multiply by \(q\) and differentiate again: \(\sum_{k=1}^\infty k^2q^{k-1}=\frac{1+q}{(1-q)^3}\). So \(E(X^2)=\frac{p(1+q)}{p^3}=\frac{1+q}{p^2}\) and
\[\operatorname{Var}(X)=\frac{1+q}{p^2}-\frac1{p^2}=\frac{q}{p^2}.\]The failures-before-success version \(X-1\) has the same variance, since shifting does not change spread.
| Concept | Essential fact |
|---|---|
| Scaling variance | \(\operatorname{Var}(aX+b)=a^2\operatorname{Var}(X)\) |
| \(X+X\) vs. independent copies | \(4\operatorname{Var}(X)\) vs. \(2\operatorname{Var}(X)\) |
| General Normal | \(X=\mu+\sigma Z\sim\mathcal N(\mu,\sigma^2)\) |
| Second parameter | Variance \(\sigma^2\); scale factor is \(\sigma\) |
| Standardization | \((X-\mu)/\sigma\sim\mathcal N(0,1)\) |
| Normal CDF | \(\Phi\left((x-\mu)/\sigma\right)\) |
| Normal PDF | \(\frac1\sigma\varphi\left((x-\mu)/\sigma\right)\) |
| Quantile | \(\mu+\sigma\,\Phi^{-1}(u)\) |
| Negation | \(-X\sim\mathcal N(-\mu,\sigma^2)\) |
| Independent sum | \(\mathcal N(\mu_1+\mu_2,\sigma_1^2+\sigma_2^2)\) |
| 68-95-99.7% | Within \(1,2,3\) SDs of the mean |
| 95% multiplier | \(1.96\) |
| LOTUS | \(E(g(X))=\sum_xg(x)P(X=x)\) |
| LOTUS proof | Regroup a sum over outcomes |
| Series trick | \(\lambda\frac{d}{d\lambda}\) brings down a factor \(k\) |
| Poisson | Mean and variance \(\lambda\) |
| Indicator facts | \(I^2=I\); \(E(I_iI_j)=P(A_i\cap A_j)\) |
| Binomial | Variance \(npq\) |
| First Success | Mean \(1/p\), variance \(q/p^2\) |
A map \(X\mapsto\mu+\sigma X\) with \(\sigma>0\). It shifts the center by \(\mu\) and stretches the spread by \(\sigma\), and it keeps a random variable within the Normal family (and within the Uniform family).
The distribution of \(\mu+\sigma Z\) for \(Z\sim\mathcal N(0,1)\), with mean \(\mu\), variance \(\sigma^2\), and PDF \(\frac1{\sigma\sqrt{2\pi}}e^{-(x-\mu)^2/(2\sigma^2)}\).
Subtracting the mean and dividing by the standard deviation, \((X-\mu)/\sigma\). For a Normal random variable the result is standard Normal.
For a Normal random variable, the approximate probabilities of lying within one, two, and three standard deviations of the mean.
\(E(g(X))=\sum_xg(x)P(X=x)\) for discrete \(X\) (with an integral against the PDF for continuous \(X\)). It follows by regrouping the sum over outcomes.
The set of outcomes on which \(X\) takes a given value \(x\), with total mass \(P(X=x)\). Grouping pebbles into super-pebbles proves LOTUS.
An expectation such as \(E(X(X-1))\). For count variables it is often easier to compute than \(E(X^2)\) and leads to the variance.
The Exponential distribution models a continuous waiting time when events occur at a constant rate \(\lambda\). Its survival function \(e^{-\lambda x}\) gives the CDF and PDF at once. Every Exponential is a rescaled \(\operatorname{Expo}(1)\), so its mean \(1/\lambda\) and variance \(1/\lambda^2\) follow from two integrals for the standard case. Its defining feature is the memoryless property: having waited any length of time without the event does not change the distribution of the remaining wait. Memorylessness gives conditional expectations such as \(E(X\mid X>a)=a+1/\lambda\) immediately, and among continuous distributions on \((0,\infty)\) only the Exponential has it.
| Symbol | Meaning |
|---|---|
| \(X\sim\operatorname{Expo}(\lambda)\) | \(X\) is Exponential with rate \(\lambda>0\) |
| \(\lambda\) | Rate: expected number of events per unit time |
| \(1/\lambda\) | Mean waiting time |
| \(f(x)=\lambda e^{-\lambda x}\) | PDF for \(x>0\) |
| \(F(x)=1-e^{-\lambda x}\) | CDF for \(x>0\) |
| \(G(x)=P(X>x)\) | Survival function, \(1-F(x)\) |
| \(s\) | Time already waited |
| \(t\) | Additional waiting time |
| \(E(X\mid A)\) | Conditional expectation of \(X\) given event \(A\) |
| \(\operatorname{Geom}(p)\) | Discrete analogue: failures before the first success |
After studying these notes, you should be able to:
A continuous random variable \(X\) has the Exponential distribution with rate \(\lambda>0\), written \(X\sim\operatorname{Expo}(\lambda)\), if its PDF is
\[f(x)=\lambda e^{-\lambda x},\qquad x>0,\]and \(f(x)=0\) for \(x\le0\).
Validity. The PDF is positive on \((0,\infty)\), and with \(u=\lambda x\),
\[\int_0^\infty\lambda e^{-\lambda x}\,dx=\int_0^\infty e^{-u}\,du=1.\]CDF and survival function. Integrating from 0 to \(x\):
The survival function is the most convenient form. It decays geometrically in continuous time: each additional unit of time multiplies the survival probability by the same factor \(e^{-\lambda}\).
Interpretation. \(X\) is the waiting time for an event that happens at constant rate \(\lambda\). If the number of events in \([0,t]\) is \(\operatorname{Pois}(\lambda t)\), then the first event time \(T\) satisfies \(\{T>t\}=\{\text{no events in }[0,t]\}\), so \(P(T>t)=e^{-\lambda t}\). Typical uses are times between arrivals in a queue, times between calls, decay times of radioactive atoms, and lifetimes of components that do not wear out.
Incorrect: In \(\operatorname{Expo}(\lambda)\), the parameter \(\lambda\) is the mean waiting time.
Correction: \(\lambda\) is a rate, in events per unit time; the mean waiting time is \(1/\lambda\). A rate of 4 calls per hour gives a mean wait of 15 minutes. Some software parametrizes the Exponential by the mean (a “scale”), so check the convention.
Let \(X\sim\operatorname{Expo}(\lambda)\) and \(Y=\lambda X\). For \(y>0\),
\[P(Y>y)=P\left(X>\frac y\lambda\right)=e^{-\lambda\cdot y/\lambda}=e^{-y}.\]Measuring time in units of the mean waiting time removes the parameter. Conversely, if \(Y\sim\operatorname{Expo}(1)\), then \(Y/\lambda\sim\operatorname{Expo}(\lambda)\). This is a scale transformation; shifting is not allowed, because it would move the support away from \((0,\infty)\).
Let \(Y\sim\operatorname{Expo}(1)\). Integrate by parts with \(u=y\) and \(dv=e^{-y}\,dy\), so \(v=-e^{-y}\):
\[E(Y)=\int_0^\infty ye^{-y}\,dy=\Big[-ye^{-y}\Big]_0^\infty+\int_0^\infty e^{-y}\,dy=0+1=1.\]By LOTUS and integration by parts with \(u=y^2\):
\[E(Y^2)=\int_0^\infty y^2e^{-y}\,dy=\Big[-y^2e^{-y}\Big]_0^\infty+2\int_0^\infty ye^{-y}\,dy=2E(Y)=2.\]So \(\operatorname{Var}(Y)=2-1^2=1\). The boundary terms vanish because \(e^{-y}\) decays faster than any power of \(y\) grows.
Write \(X=Y/\lambda\) and use \(E(cY)=cE(Y)\) and \(\operatorname{Var}(cY)=c^2\operatorname{Var}(Y)\):
The mean and standard deviation are equal. A faster rate gives a shorter and less variable wait.
A sensor has lifetime \(T\sim\operatorname{Expo}(\lambda)\) with mean 1000 hours, so \(\lambda=0.001\) per hour.
(a) Survival past 1500 hours.
\[P(T>1500)=e^{-0.001(1500)}=e^{-1.5}\approx0.223.\](b) Median lifetime. Solve \(e^{-\lambda m}=\tfrac12\):
\[m=\frac{\ln2}{\lambda}\approx693\text{ hours}.\]The median is below the mean because the distribution is right-skewed: a few very long lifetimes pull the mean up.
(c) Probability of failing before the mean. \(P(T<1000)=1-e^{-1}\approx0.632\), not \(0.5\).
A distribution on \((0,\infty)\) is memoryless if a random variable \(X\) with that distribution satisfies
\[P(X\ge s+t\mid X\ge s)=P(X\ge t)\quad\text{for all }s,t>0.\]Here \(s\) is the time already waited and \(t\) is an additional wait. The condition says that, given no event by time \(s\), the remaining waiting time \(X-s\) has the same distribution as a fresh wait \(X\).
The Exponential is memoryless. Since \(\{X\ge s+t\}\subseteq\{X\ge s\}\), the intersection in the conditional probability is just \(\{X\ge s+t\}\):
\[P(X\ge s+t\mid X\ge s)=\frac{P(X\ge s+t)}{P(X\ge s)}=\frac{e^{-\lambda(s+t)}}{e^{-\lambda s}}=e^{-\lambda t}=P(X\ge t).\]The key step is \(e^{-\lambda(s+t)}=e^{-\lambda s}e^{-\lambda t}\): the survival function turns sums of times into products of probabilities.
Exponential wait. A server’s next request arrives after \(X\sim\operatorname{Expo}(1/2)\) seconds, mean 2 seconds. Given no request in the first 5 seconds, the probability that the next 2 seconds are also quiet is
\[P(X\ge7\mid X\ge5)=e^{-(1/2)(2)}=e^{-1}\approx0.368,\]the same as for a fresh 2-second window. The 5 seconds already waited are irrelevant.
Uniform wait. A shuttle arrives at a time \(W\sim\operatorname{Unif}(0,20)\) minutes. Unconditionally, \(P(W>4)=16/20=0.8\). Given that 12 minutes have passed, only 8 minutes remain possible, so
\[P(W>16\mid W>12)=\frac{P(W>16)}{P(W>12)}=\frac{4/20}{8/20}=0.5.\]Waiting makes the shuttle more likely to arrive soon. The Uniform is not memoryless.
When the assumption fits. Radioactive decay is well modeled as memoryless: an atom that has not decayed is physically identical to a fresh one. Machine and human lifetimes are not, because ageing makes failure more likely as time passes. The Exponential is still important for such problems, as a building block for more flexible lifetime models and because of its exact links to the Poisson.
Incorrect: Memoryless means the total waiting time does not depend on how long one has already waited, so \(P(X\ge s+t\mid X\ge s)=P(X\ge s+t)\).
Correction: Memorylessness concerns the remaining wait. Given \(X\ge s\), the remaining time \(X-s\) is distributed like a fresh \(X\). The total time \(X\) is still at least \(s\), so conditioning does change the distribution of \(X\) itself.
Let \(X\sim\operatorname{Expo}(\lambda)\) and \(a>0\). Find \(E(X\mid X>a)\).
Write \(X=a+(X-a)\). Given \(X>a\), memorylessness says \(X-a\sim\operatorname{Expo}(\lambda)\). Conditional expectation is linear, like ordinary expectation, so
The time already spent is added to a fresh expected wait. For the sensor with mean life 1000 hours that has already run 500 hours, the expected total life is 1500 hours and the expected remaining life is still 1000 hours.
Conditional expectation \(E(X\mid A)\) is computed like \(E(X)\), with the conditional distribution of \(X\) given \(A\) in place of the unconditional one. It is treated systematically later in the course.
Given \(X>a\), the conditional PDF is the original PDF restricted to \((a,\infty)\) and rescaled by \(P(X>a)\):
\[f(x\mid X>a)=\frac{\lambda e^{-\lambda x}}{e^{-\lambda a}}=\lambda e^{-\lambda(x-a)},\qquad x>a.\]This is an \(\operatorname{Expo}(\lambda)\) PDF shifted to start at \(a\). Substituting \(y=x-a\),
\[E(X\mid X>a)=\int_a^\infty x\lambda e^{-\lambda(x-a)}\,dx=\int_0^\infty(y+a)\lambda e^{-\lambda y}\,dy=\frac1\lambda+a.\]If \(X\) is a positive continuous random variable with the memoryless property, then \(X\sim\operatorname{Expo}(\lambda)\) for some \(\lambda>0\).
Supplementary proof by differentiation. The main proof, which needs only continuity, is in Lecture 17. A shorter argument, assuming differentiability, runs as follows. Let \(G(x)=P(X>x)\). For a continuous \(X\), \(P(X\ge x)=P(X>x)\), so memorylessness reads
\[G(s+t)=G(s)G(t)\quad\text{for all }s,t>0.\]Since \(X\) is continuous, \(G\) is differentiable except possibly at finitely many points; assume differentiability for the argument. Differentiate with respect to \(s\), then let \(s\to0\):
\[G'(t)=G'(0)G(t).\]Set \(c=G'(0)\), a constant. The equation \(G'=cG\) with \(G(0)=P(X>0)=1\) has the unique solution \(G(t)=e^{ct}\). Because \(G\) decreases toward 0, \(c<0\). Writing \(\lambda=-c>0\) gives \(G(t)=e^{-\lambda t}\), the Exponential survival function. \(\blacksquare\)
The Exponential is the continuous counterpart of the Geometric distribution.
| Feature | \(\operatorname{Geom}(p)\) | \(\operatorname{Expo}(\lambda)\) |
|---|---|---|
| Waiting for | First success in discrete trials | First event in continuous time |
| Support | \(\{0,1,2,\ldots\}\) | \((0,\infty)\) |
| Survival | \(P(X\ge k)=q^k\) | \(P(X>x)=e^{-\lambda x}\) |
| Mean | \(q/p\) | \(1/\lambda\) |
| Memoryless | Yes, among distributions on \(\{0,1,\ldots\}\) | Yes, among continuous distributions on \((0,\infty)\) |
Discrete memorylessness. For \(X\sim\operatorname{Geom}(p)\), \(P(X\ge m+n\mid X\ge m)=q^{m+n}/q^m=q^n=P(X\ge n)\). A run of failures does not make a success more likely: independent trials have no memory.
Supplementary: a limit. Perform trials every \(1/n\) time units with success probability \(\lambda/n\). The number of failures before the first success is \(\operatorname{Geom}(\lambda/n)\), and the waiting time in time units is that count divided by \(n\). Its survival probability at time \(x\) is approximately \((1-\lambda/n)^{nx}\to e^{-\lambda x}\), the same compound-interest limit as in the Poisson limit of the Binomial.
Try these before reading the answers.
\(P(X>0.5)=e^{-2}\approx0.135\). Median \(\ln2/4\approx0.173\). \(E(X)=\operatorname{SD}(X)=0.25\).
\(P(3Y>x)=P(Y>x/3)=e^{-x/3}\), so \(3Y\sim\operatorname{Expo}(1/3)\), with mean 3.
\(E(e^{X/2})=\int_0^\infty e^{x/2}e^{-x}\,dx=\int_0^\infty e^{-x/2}\,dx=2\). For \(E(e^X)\) the integrand is \(e^{x}e^{-x}=1\), and \(\int_0^\infty1\,dx=\infty\): the expectation is infinite.
Rate \(\lambda=6\) per hour, so 10 minutes is \(1/6\) hour. By memorylessness the past 20 minutes are irrelevant: \(P=e^{-6/6}=e^{-1}\approx0.368\). Expected time to the next call is \(1/6\) hour, 10 minutes.
\(P(U\ge0.75\mid U\ge0.5)=\frac{0.25}{0.5}=0.5\), while \(P(U\ge0.25)=0.75\). They differ.
\(E(X\mid X>2)=3\). With \(P(X>2)=e^{-2}\) and \(E(X)=1\):
\[E(X\mid X<2)=\frac{1-3e^{-2}}{1-e^{-2}}\approx0.687.\]It lies between 0 and 2 and below 1, as it must.
\(e^{-5730\lambda}=\tfrac12\) gives \(\lambda=\ln2/5730\approx1.21\times10^{-4}\) per year. Mean lifetime \(1/\lambda=5730/\ln2\approx8267\) years, longer than the half-life.
\(P(X>2/\lambda)=e^{-\lambda\cdot2/\lambda}=e^{-2}\approx0.135\). Any statement in units of the mean is the same for all Exponentials, by the scaling result.
Memorylessness would give the 3000-hour bulb another 1000 expected hours, exactly as for a new bulb. Real filaments degrade, so an old bulb is more likely to fail soon. A lifetime model with an increasing failure rate is more realistic.
| Concept | Essential fact |
|---|---|
| \(\lambda e^{-\lambda x}\), \(x>0\) | |
| CDF | \(1-e^{-\lambda x}\) |
| Survival | \(P(X>x)=e^{-\lambda x}\) |
| Rate vs. mean | Rate \(\lambda\), mean \(1/\lambda\) |
| Scaling | \(\lambda X\sim\operatorname{Expo}(1)\) |
| \(\operatorname{Expo}(1)\) moments | \(E(Y)=1\), \(E(Y^2)=2\), \(\operatorname{Var}(Y)=1\) |
| General moments | Mean \(1/\lambda\), variance \(1/\lambda^2\) |
| Median | \(\ln2/\lambda\) |
| Memoryless | \(P(X\ge s+t\mid X\ge s)=P(X\ge t)\) |
| Remaining wait | Given \(X>s\), \(X-s\sim\operatorname{Expo}(\lambda)\) |
| Conditional mean | \(E(X\mid X>a)=a+1/\lambda\) |
| Characterization | Only memoryless continuous distribution on \((0,\infty)\) |
| Functional equation | \(G(s+t)=G(s)G(t)\) |
| Discrete analogue | Geometric, also memoryless |
The continuous distribution on \((0,\infty)\) with PDF \(\lambda e^{-\lambda x}\), modeling the waiting time for an event occurring at constant rate \(\lambda\).
The parameter \(\lambda\) of the Exponential, measured in events per unit time. It is the reciprocal of the mean waiting time.
\(G(x)=P(X>x)=1-F(x)\). For the Exponential it is \(e^{-\lambda x}\).
\(P(X\ge s+t\mid X\ge s)=P(X\ge t)\) for all \(s,t>0\): the remaining wait does not depend on the time already waited.
\(E(X\mid A)\), the mean of \(X\) computed with its conditional distribution given \(A\).
The median of an Exponential lifetime, \(\ln2/\lambda\): the time by which half of a large population of independent items has decayed or failed.
Memorylessness is rarely true of lifetimes, but it clarifies what a life expectancy is: a conditional expectation given survival to a certain age, not one fixed number. The moment generating function \(M(t)=E(e^{tX})\) packages every moment of a distribution into one function. Its derivatives at \(t=0\) are the moments, it determines the distribution uniquely, and it turns sums of independent random variables into products. These properties give the Binomial MGF from the Bernoulli MGF in one line and the Normal MGF by completing the square. Laplace’s rule of succession treats an unknown success probability as a Uniform random variable and updates it with data: after \(n\) successes in \(n\) trials, the probability of another success is \((n+1)/(n+2)\).
| Symbol | Meaning |
|---|---|
| \(M_X(t)=E(e^{tX})\) | Moment generating function of \(X\) |
| \(t\) | Dummy variable of the MGF; no physical meaning |
| \((-a,a)\) | Open interval around 0 on which the MGF must be finite |
| \(E(X^n)\) | \(n\)th moment of \(X\) |
| \(M^{(n)}(0)\) | \(n\)th derivative of \(M\) evaluated at \(t=0\) |
| \(E(T\mid T>a)\) | Expected lifetime given survival to age \(a\) |
| \(p\sim\operatorname{Unif}(0,1)\) | Prior distribution for an unknown success probability |
| \(S_n=X_1+\cdots+X_n\) | Number of successes in the first \(n\) trials |
| \(f(p\mid S_n=k)\) | Posterior density of \(p\) given the data |
After studying these notes, you should be able to:
Lecture 16 showed that the Exponential is the only memoryless continuous distribution on \((0,\infty)\). For a memoryless lifetime \(T\sim\operatorname{Expo}(\lambda)\), the expected remaining life at every age is \(1/\lambda\), so
\[E(T\mid T>a)=a+\frac1\lambda.\]Let \(X\) be a positive continuous random variable with survival function \(G(x)=P(X\ge x)\). Memorylessness says \(P(X\ge s+t)=P(X\ge s)P(X\ge t)\), that is,
\[G(s+t)=G(s)G(t)\quad\text{for all }s,t>0.\]The strategy is to solve this equation for the unknown function \(G\).
This is the \(\operatorname{Expo}(\lambda)\) survival function. \(\blacksquare\)
This argument needs only the continuity of \(G\), not differentiability; a shorter supplementary proof that assumes differentiability is included in Lecture 16. (Step 5 also uses \(0<G(1)<1\): \(G(1)=0\) or \(G(1)=1\) would put all probability at 0 or at infinity, impossible for a positive continuous random variable.)
Human lifetimes are not memoryless: an 80-year-old does not have the remaining life expectancy of a newborn. But the same conditional quantity is what answers practical questions about life expectancy.
Life expectancy is conditional. A single number such as “life expectancy at birth is 80 years” is \(E(T)\), averaged over all lifetimes, including those that end early. Someone who has reached age \(a\) has survived the early risks, so their expected lifetime is \(E(T\mid T>a)\), which is at least \(a\) and typically larger than \(E(T)\).
Take a deliberately simple lifetime model \(T\sim\operatorname{Unif}(0,100)\) years.
Conditional expected total life increases with age, while expected remaining life decreases. Under an Exponential model the expected remaining life would instead stay constant.
Censored data. Estimating lifetimes from data has an extra difficulty: some subjects are still alive when a study ends, so only a lower bound \(T>c\) is observed. Such observations are called censored. Ignoring them, or treating \(c\) as the lifetime, biases estimates downward. Survival analysis develops methods for this; the conditional reasoning above is its starting point.
Incorrect: If life expectancy at birth is 80, a 79-year-old should expect to live about one more year.
Correction: That uses the unconditional mean for someone already known to have survived to 79. The relevant quantity is \(E(T\mid T>79)\), which exceeds 80 by more than one year in realistic models.
The moment generating function (MGF) of a random variable \(X\) is
\[M_X(t)=E\left(e^{tX}\right),\]as a function of \(t\), provided it is finite for all \(t\) in some open interval \((-a,a)\) with \(a>0\). Otherwise \(X\) has no MGF.
What \(t\) is. \(t\) has no probabilistic meaning. It is a bookkeeping variable that lets calculus act on the whole sequence of moments at once.
Why the name. Expand \(e^{tX}\) as a power series and take expectations term by term (justified when the MGF exists on an interval around 0):
\[M(t)=E\left(\sum_{n=0}^\infty\frac{X^nt^n}{n!}\right)=\sum_{n=0}^\infty E(X^n)\frac{t^n}{n!}.\]The MGF is a power series whose coefficients are the moments, each divided by \(n!\).
A built-in check. \(M(0)=E(e^0)=1\) for every MGF.
The MGF is another description of the distribution. Alongside the CDF and the PMF or PDF, it is a third way to encode a distribution. It is computed with LOTUS: \(M(t)=\sum_xe^{tx}P(X=x)\) or \(\int e^{tx}f(x)\,dx\).
Proof. A power series equals its own Taylor series, so the coefficient of \(t^n/n!\) in \(M(t)\) is \(M^{(n)}(0)\). From the expansion in Part 2 the same coefficient is \(E(X^n)\). Matching coefficients gives the result. \(\blacksquare\)
Directly: \(M'(t)=E(Xe^{tX})\), so \(M'(0)=E(X)\); \(M''(t)=E(X^2e^{tX})\), so \(M''(0)=E(X^2)\). Differentiation replaces an integral or sum for each moment.
If \(X\) and \(Y\) have MGFs that agree on some open interval around 0, then \(X\) and \(Y\) have the same distribution. The proof is a difficult result in analysis and is not given here. Its use is identification: compute an MGF, recognize its form, and conclude the distribution.
If \(X\) and \(Y\) are independent, then
The middle step uses that \(e^{tX}\) and \(e^{tY}\) are independent (functions of independent variables) and that the expectation of a product of independent random variables is the product of their expectations, proved later in the course. Finding the PMF or PDF of a sum directly requires a convolution sum or integral; with MGFs it is a multiplication.
Incorrect: \(M_{X+Y}=M_X\,M_Y\) for any \(X\) and \(Y\).
Correction: Independence is required. For example, with \(Y=X\), \(M_{2X}(t)=M_X(2t)\), which generally differs from \(M_X(t)^2\).
For constants \(a,b\),
\[M_{a+bX}(t)=E\left(e^{t(a+bX)}\right)=e^{at}E\left(e^{(bt)X}\right)=e^{at}M_X(bt).\]A shift multiplies the MGF by \(e^{at}\); a scale rescales its argument.
Bernoulli. If \(X\sim\operatorname{Bern}(p)\), then \(e^{tX}\) equals \(e^t\) with probability \(p\) and \(e^0=1\) with probability \(q\):
\[M_X(t)=pe^t+q,\qquad t\in\mathbb R.\]Check: \(M(0)=p+q=1\), and \(M'(0)=p=E(X)\).
Binomial. \(X\sim\operatorname{Bin}(n,p)\) is a sum of \(n\) i.i.d. \(\operatorname{Bern}(p)\) variables. By the sum property,
\[M_X(t)=\left(pe^t+q\right)^n.\]Moments. Differentiate once:
\[M'(t)=n\left(pe^t+q\right)^{n-1}pe^t\implies M'(0)=np.\]Differentiate again with the product rule:
\[M''(t)=n(n-1)\left(pe^t+q\right)^{n-2}p^2e^{2t}+n\left(pe^t+q\right)^{n-1}pe^t\implies M''(0)=n(n-1)p^2+np.\]So \(\operatorname{Var}(X)=n(n-1)p^2+np-n^2p^2=np(1-p)\), matching Lecture 14.
A sum of Binomials. If \(X\sim\operatorname{Bin}(n,p)\) and \(Y\sim\operatorname{Bin}(m,p)\) are independent, then \(M_{X+Y}(t)=(pe^t+q)^{n+m}\), the \(\operatorname{Bin}(n+m,p)\) MGF. By uniqueness, \(X+Y\sim\operatorname{Bin}(n+m,p)\).
Let \(Z\sim\mathcal N(0,1)\). By LOTUS,
\[M_Z(t)=\frac1{\sqrt{2\pi}}\int_{-\infty}^\infty e^{tz}e^{-z^2/2}\,dz=\frac1{\sqrt{2\pi}}\int_{-\infty}^\infty e^{-(z^2-2tz)/2}\,dz.\]Complete the square in the exponent: \(z^2-2tz=(z-t)^2-t^2\). Then
\[M_Z(t)=e^{t^2/2}\int_{-\infty}^\infty\frac1{\sqrt{2\pi}}e^{-(z-t)^2/2}\,dz=e^{t^2/2}.\]The remaining integrand is the \(\mathcal N(t,1)\) PDF, which integrates to 1.
The general case follows from the location-scale rule: \(M_X(t)=e^{\mu t}M_Z(\sigma t)\).
Checks. \(M_X'(t)=(\mu+\sigma^2t)M_X(t)\) gives \(M_X'(0)=\mu\). \(M_X''(t)=\left(\sigma^2+(\mu+\sigma^2t)^2\right)M_X(t)\) gives \(M_X''(0)=\sigma^2+\mu^2\), so the variance is \(\sigma^2\).
Sums of independent Normals. If \(X_j\sim\mathcal N(\mu_j,\sigma_j^2)\) are independent, then
\[M_{X_1+X_2}(t)=e^{(\mu_1+\mu_2)t+(\sigma_1^2+\sigma_2^2)t^2/2},\]the \(\mathcal N(\mu_1+\mu_2,\sigma_1^2+\sigma_2^2)\) MGF. This proves the result stated in Lecture 14.
Expand \(e^{t^2/2}=\sum_{k=0}^\infty\frac{t^{2k}}{2^kk!}\) and match with \(\sum_nE(Z^n)\frac{t^n}{n!}\). Odd moments are 0, and
\[E(Z^{2k})=\frac{(2k)!}{2^kk!}=1\cdot3\cdot5\cdots(2k-1).\]So \(E(Z^2)=1\), \(E(Z^4)=3\), \(E(Z^6)=15\), with no further integration.
An experiment has been repeated \(n\) times with \(n\) successes. What is the probability that the next trial is also a success? Laplace’s example was the sun rising: given that the sun has risen on each of \(n\) recorded days, how probable is a sunrise tomorrow?
Model. The success probability \(p\) is unknown, so treat it as a random variable.
The trials are conditionally independent given \(p\) but not unconditionally independent: each success raises the plausibility of large \(p\) and hence of further successes.
Posterior. Bayes’ rule with a density for \(p\) replaces the prior probability with the prior PDF. With \(k\) successes in \(n\) trials, the likelihood is \(\binom nkp^k(1-p)^{n-k}\), so in general
\[f(p\mid S_n=k)\propto p^k(1-p)^{n-k},\qquad0<p<1,\]where \(\propto\) means “proportional to, as a function of \(p\)”: constants not involving \(p\) are absorbed into the normalizing constant. For the sunrise case \(k=n\):
\[f(p\mid S_n=n)=\frac{P(S_n=n\mid p)\,f(p)}{P(S_n=n)}=\frac{p^n\cdot1}{P(S_n=n)},\qquad0<p<1.\]The denominator does not depend on \(p\); it is the normalizing constant. By the law of total probability with a density,
\[P(S_n=n)=\int_0^1p^n\,dp=\frac1{n+1}.\]So the posterior density is
\[f(p\mid S_n=n)=(n+1)p^n,\qquad0<p<1.\]Prediction. Condition on \(p\) and average over its posterior:
\[P(X_{n+1}=1\mid S_n=n)=\int_0^1p\cdot(n+1)p^n\,dp=\frac{n+1}{n+2}.\]The prediction is the mean of the posterior distribution of \(p\). This is the fundamental bridge in conditional form: given \(p\), the probability of success is \(p\), so averaging over what is now believed about \(p\) gives \(P(X_{n+1}=1\mid S_n=n)=E(p\mid S_n=n)\).
| Observed | Prediction for next trial |
|---|---|
| No data (\(n=0\)) | \(1/2\), the prior mean |
| 1 success in 1 trial | \(2/3\) |
| 10 successes in 10 trials | \(11/12\approx0.917\) |
| 1000 successes in 1000 trials | \(1001/1002\approx0.999\) |
The prediction approaches 1 but never reaches it. Compare the naive estimate \(n/n=1\), which would rule out failure entirely after any streak of successes.
Limits of the model. The rule assumes exchangeable trials with an unchanging \(p\) and a Uniform prior. For sunrises, physics supplies far stronger information than a Uniform prior, and days are not repeated independent trials. The rule is a model of reasoning from repetition alone.
With \(k\) successes in \(n\) trials, the posterior is proportional to \(p^k(1-p)^{n-k}\). Using the integral \(\int_0^1p^a(1-p)^b\,dp=\frac{a!\,b!}{(a+b+1)!}\) (derived later in the course), the predictive probability of success is
\[P(X_{n+1}=1\mid S_n=k)=\frac{k+1}{n+2}.\]It behaves like adding one imaginary success and one imaginary failure to the data.
Incorrect: If \(X_1,X_2,\ldots\) are i.i.d. given \(p\), they are independent, so past successes cannot change the probability of the next success.
Correction: Conditional independence given \(p\) is not independence. When \(p\) is unknown, observing successes changes the distribution of \(p\), which changes the predictive probability of the next trial.
Try these before reading the answers.
\(M(t)=\frac{e^t+e^{2t}+e^{3t}}3\). Then \(M'(0)=\frac{1+2+3}3=2\) and \(M''(0)=\frac{1+4+9}3=\frac{14}3\), so \(\operatorname{Var}(X)=\frac{14}3-4=\frac23\).
The form \((pe^t+q)^n\) with \(p=0.3\), \(n=8\) is the \(\operatorname{Bin}(8,0.3)\) MGF. By uniqueness, \(X\sim\operatorname{Bin}(8,0.3)\), with variance \(8(0.3)(0.7)=1.68\).
Match \(e^{\mu t+\sigma^2t^2/2}\): \(\mu=2\) and \(\sigma^2/2=8\), so \(\sigma^2=16\). \(X\sim\mathcal N(2,16)\) and \(P(X>10)=1-\Phi(2)\approx0.0228\).
\(M_Y(t)=e^{3t}M_Z(-2t)=e^{3t+2t^2}\), the \(\mathcal N(3,4)\) MGF.
\(M_{1-X}(t)=e^tM_X(-t)=e^t\left(pe^{-t}+q\right)=p+qe^t\), the \(\operatorname{Bern}(q)\) MGF.
\(E(X)\) is finite, but for any \(t>0\), \(e^{tx}f(x)=2e^{tx}x^{-3}\to\infty\), so \(E(e^{tX})=\infty\). The MGF is not finite on any interval around 0.
Given \(T>30\), \(T\sim\operatorname{Unif}(30,90)\), so \(E(T\mid T>30)=60\). Expected remaining life is 30 years, compared with \(E(T)=45\) at birth.
After 20 successes: \(21/22\approx0.955\). After 0 successes in \(n\) trials, the posterior is \((n+1)(1-p)^n\) and the predictive probability is \(\int_0^1p(n+1)(1-p)^n\,dp=\frac1{n+2}\).
\(\frac{7+1}{10+2}=\frac23\approx0.667\), pulled slightly from \(0.7\) toward the prior mean \(1/2\).
| Concept | Essential fact |
|---|---|
| Life expectancy | \(E(T\mid T>a)\) depends on current age \(a\) |
| Memoryless lifetime | \(E(T\mid T>a)=a+1/\lambda\) |
| MGF | \(M(t)=E(e^{tX})\), finite near 0 |
| Check | \(M(0)=1\) |
| Series form | \(M(t)=\sum_nE(X^n)t^n/n!\) |
| Moments | \(E(X^n)=M^{(n)}(0)\) |
| Uniqueness | Same MGF near 0 \(\implies\) same distribution |
| Independent sums | \(M_{X+Y}=M_XM_Y\) |
| Location-scale | \(M_{a+bX}(t)=e^{at}M_X(bt)\) |
| Bernoulli | \(pe^t+q\) |
| Binomial | \((pe^t+q)^n\) |
| Standard Normal | \(e^{t^2/2}\) |
| Normal | \(e^{\mu t+\sigma^2t^2/2}\) |
| Exponential (supplementary) | \(\lambda/(\lambda-t)\), \(t<\lambda\) |
| Poisson (supplementary) | \(e^{\lambda(e^t-1)}\) |
| Rule of succession | \((n+1)/(n+2)\); generally \((k+1)/(n+2)\) |
\(E(T\mid T>a)\), the expected lifetime of someone known to have survived to age \(a\).
An observation known only to exceed (or fall below) some value, such as a subject still alive at the end of a study.
\(M_X(t)=E(e^{tX})\), required to be finite on an open interval around 0. Its Taylor coefficients encode all moments of \(X\).
\(E(X^n)\) for a positive integer \(n\). The first moment is the mean; the first two determine the variance.
Rewriting \(z^2-2tz\) as \((z-t)^2-t^2\), which turns the Normal MGF integral into a Normal PDF times a constant.
The distribution of an unknown parameter before observing data, and its conditional distribution after the data, linked by Bayes’ rule.
Independence that holds given the value of another random variable, such as Bernoulli trials given \(p\). It does not imply unconditional independence.
Under a Uniform prior on \(p\), the probability of success on the next trial after \(k\) successes in \(n\) trials is \((k+1)/(n+2)\).
When an MGF has a recognizable power series, every moment can be read off at once by matching coefficients. This gives \(E(X^n)=n!/\lambda^n\) for the Exponential and \(E(Z^{2k})=(2k-1)(2k-3)\cdots1\) for the standard Normal. The Poisson MGF \(e^{\lambda(e^t-1)}\) shows in one line that independent Poissons add to a Poisson. The second half moves from one random variable to several. A joint distribution describes random variables together; marginal distributions are obtained by summing or integrating out the others; conditional distributions renormalize a slice of the joint. Independence means the joint factors into marginals everywhere. A Uniform point in a square has independent coordinates, while a Uniform point in a disk does not.
| Symbol | Meaning |
|---|---|
| \(M(t)=E(e^{tX})\) | Moment generating function |
| \(E(X^n)\) | \(n\)th moment |
| \((2k-1)!!\) | Double factorial \((2k-1)(2k-3)\cdots3\cdot1\) |
| \(F_{X,Y}(x,y)=P(X\le x,Y\le y)\) | Joint CDF |
| \(P(X=x,Y=y)\) | Joint PMF (discrete) |
| \(f_{X,Y}(x,y)\) | Joint PDF (continuous) |
| \(P(X=x)=\sum_yP(X=x,Y=y)\) | Marginal PMF |
| \(f_X(x)=\int f_{X,Y}(x,y)\,dy\) | Marginal PDF |
| \(P(Y=y\mid X=x)\), \(f_{Y\mid X}(y\mid x)\) | Conditional PMF, conditional PDF |
| \(\iint_Af_{X,Y}\,dx\,dy\) | Probability that \((X,Y)\) falls in region \(A\) |
After studying these notes, you should be able to:
The name comes from mechanics. Place mass \(P(X=x)\) at each point \(x\) of a rod, or spread mass with density \(f\).
Higher moments \(E(X^n)\) extend the same weighted-power idea. The MGF collects all of them in one function, \(M(t)=\sum_nE(X^n)t^n/n!\), when it exists near 0.
Two ways to extract moments from an MGF:
Let \(X\sim\operatorname{Expo}(1)\). For \(t<1\),
\[M(t)=\int_0^\infty e^{tx}e^{-x}\,dx=\int_0^\infty e^{-(1-t)x}\,dx=\frac1{1-t}.\]For \(t\ge1\) the integral diverges, but the MGF is finite on \((-1,1)\), which is all that is needed.
Match coefficients. For \(\lvert t\rvert<1\), the geometric series gives
\[\frac1{1-t}=\sum_{n=0}^\infty t^n=\sum_{n=0}^\infty n!\,\frac{t^n}{n!}.\]Comparing with \(M(t)=\sum_nE(X^n)\frac{t^n}{n!}\):
General rate. If \(Y\sim\operatorname{Expo}(\lambda)\), then \(Y=X/\lambda\) with \(X\sim\operatorname{Expo}(1)\), so
\[E(Y^n)=\frac{E(X^n)}{\lambda^n}=\frac{n!}{\lambda^n},\qquad M_Y(t)=M_X(t/\lambda)=\frac{\lambda}{\lambda-t},\quad t<\lambda.\]In particular \(E(Y)=1/\lambda\) and \(\operatorname{Var}(Y)=2/\lambda^2-1/\lambda^2=1/\lambda^2\), confirming Lecture 16 without integration by parts.
For \(Y\sim\operatorname{Expo}(2)\), the third and tenth moments are
\[E(Y^3)=\frac{3!}{2^3}=\frac68=0.75,\qquad E(Y^{10})=\frac{10!}{2^{10}}=\frac{3{,}628{,}800}{1024}\approx3543.75.\]Differentiating \(\frac{2}{2-t}\) ten times would give the same answer with much more work.
Let \(Z\sim\mathcal N(0,1)\), with \(M(t)=e^{t^2/2}\) (Lecture 17). Expand the exponential series with \(t^2/2\) in place of the variable:
\[e^{t^2/2}=\sum_{k=0}^\infty\frac{(t^2/2)^k}{k!}=\sum_{k=0}^\infty\frac{t^{2k}}{2^kk!}=\sum_{k=0}^\infty\frac{(2k)!}{2^kk!}\cdot\frac{t^{2k}}{(2k)!}.\]Only even powers of \(t\) appear, so every odd moment is 0, as symmetry requires. The even moments are the coefficients of \(t^{2k}/(2k)!\):
Why the two forms agree. \((2k)!\) is the product of the numbers \(1,\ldots,2k\). Its even factors are \(2\cdot4\cdots2k=2^kk!\); dividing them out leaves the odd factors.
| \(n\) | 2 | 4 | 6 | 8 |
|---|---|---|---|---|
| \(E(Z^n)\) | 1 | 3 | 15 | 105 |
The fourth moment 3 is the benchmark for tail heaviness: a distribution with standardized fourth moment above 3 has heavier tails than the Normal.
Poisson MGF. For \(X\sim\operatorname{Pois}(\lambda)\), by LOTUS and the exponential series,
\[M_X(t)=\sum_{k=0}^\infty e^{tk}\frac{e^{-\lambda}\lambda^k}{k!}=e^{-\lambda}\sum_{k=0}^\infty\frac{(\lambda e^t)^k}{k!}=e^{-\lambda}e^{\lambda e^t}=e^{\lambda(e^t-1)}.\]Moments check. \(M'(t)=\lambda e^tM(t)\), so \(E(X)=\lambda\). \(M''(t)=(\lambda e^t+\lambda^2e^{2t})M(t)\), so \(E(X^2)=\lambda+\lambda^2\) and \(\operatorname{Var}(X)=\lambda\).
Sum. If \(X\sim\operatorname{Pois}(\lambda)\) and \(Y\sim\operatorname{Pois}(\mu)\) are independent,
\[M_{X+Y}(t)=e^{\lambda(e^t-1)}e^{\mu(e^t-1)}=e^{(\lambda+\mu)(e^t-1)},\]the \(\operatorname{Pois}(\lambda+\mu)\) MGF. By uniqueness:
This matches the story: two independent streams of rare events, at rates \(\lambda\) and \(\mu\), merge into one stream at rate \(\lambda+\mu\).
Incorrect: If \(X\) and \(Y\) are each \(\operatorname{Pois}(\lambda)\), then \(X+Y\sim\operatorname{Pois}(2\lambda)\).
Correction: Independence is essential. If \(Y=X\), then \(X+Y=2X\) takes only even values, so it cannot be Poisson. Its MGF is \(M_X(2t)=e^{\lambda(e^{2t}-1)}\), not \(e^{2\lambda(e^t-1)}\).
One random variable is described by its CDF, PMF, or PDF. Several random variables defined on the same experiment need a description of how they behave together.
For random variables \(X\) and \(Y\):
How they relate. The joint determines both marginals and all conditionals. The marginals alone do not determine the joint, because they discard how the variables move together. The joint can be rebuilt from a marginal and a conditional: \(f_{X,Y}(x,y)=f_{Y\mid X}(y\mid x)f_X(x)\).
Independence. \(X\) and \(Y\) are independent exactly when the joint factors into the marginals for all \(x,y\):
\[P(X=x,Y=y)=P(X=x)P(Y=y)\quad\text{or}\quad f_{X,Y}(x,y)=f_X(x)f_Y(y).\]Equivalently, every conditional distribution of \(Y\) given \(X=x\) equals the marginal distribution of \(Y\).
Let \(X\in\{0,1\}\) and \(Y\in\{0,1,2\}\) have the joint PMF below.
| \(y=0\) | \(y=1\) | \(y=2\) | \(P(X=x)\) | |
|---|---|---|---|---|
| \(x=0\) | 0.10 | 0.20 | 0.10 | 0.40 |
| \(x=1\) | 0.20 | 0.30 | 0.10 | 0.60 |
| \(P(Y=y)\) | 0.30 | 0.50 | 0.20 | 1 |
Marginals. Row sums give the PMF of \(X\); column sums give the PMF of \(Y\). They are written in the margins of the table, which is where the name comes from.
Independence check. \(P(X=0,Y=0)=0.10\), but \(P(X=0)P(Y=0)=0.4\times0.3=0.12\). One failing cell is enough: \(X\) and \(Y\) are dependent.
Conditional distribution. Given \(X=1\), divide the second row by \(0.60\):
\[P(Y=0\mid X=1)=\frac13,\quad P(Y=1\mid X=1)=\frac12,\quad P(Y=2\mid X=1)=\frac16.\]Its mean is \(E(Y\mid X=1)=0\cdot\frac13+1\cdot\frac12+2\cdot\frac16=\frac56\), compared with \(E(Y)=0.9\) overall.
An independent table with the same marginals. Filling each cell with the product of its margins, for example \(0.4\times0.3=0.12\) in the top-left, gives a different joint PMF with identical marginals.
Both tables below describe a pair of Bernoulli random variables \(X\) and \(Y\).
| Table A | \(y=0\) | \(y=1\) | \(P(X=x)\) |
|---|---|---|---|
| \(x=0\) | 1/3 | 1/6 | 1/2 |
| \(x=1\) | 1/3 | 1/6 | 1/2 |
| \(P(Y=y)\) | 2/3 | 1/3 | 1 |
| Table B | \(y=0\) | \(y=1\) | \(P(X=x)\) |
|---|---|---|---|
| \(x=0\) | 1/3 | 0 | 1/3 |
| \(x=1\) | 1/3 | 1/3 | 2/3 |
| \(P(Y=y)\) | 2/3 | 1/3 | 1 |
Table A is independent: every cell equals the product of its margins, for example \(\frac12\cdot\frac23=\frac13\) and \(\frac12\cdot\frac13=\frac16\). Both rows are the same, so learning \(X\) does not change the distribution of \(Y\).
Table B is dependent: \(P(X=0,Y=1)=0\) but \(P(X=0)P(Y=1)=\frac13\cdot\frac13=\frac19\). Here \(Y=1\) forces \(X=1\). The marginal of \(Y\) is the same in both tables; only the joint reveals the difference.
Dependent. Let \(f(x,y)=x+y\) on the unit square \(0<x,y<1\).
Independent. Let \(f(x,y)=6e^{-2x-3y}\) for \(x,y>0\). It factors as \(\left(2e^{-2x}\right)\left(3e^{-3y}\right)\) on a product region, so \(X\sim\operatorname{Expo}(2)\) and \(Y\sim\operatorname{Expo}(3)\) are independent.
A Uniform point in a region \(R\) of the plane has joint PDF equal to \(1/\text{area}(R)\) on \(R\) and 0 elsewhere. Probabilities are then proportional to area.
Square. For the unit square, \(f_{X,Y}(x,y)=1\) for \(0\le x,y\le1\). This equals \(1\cdot1=f_X(x)f_Y(y)\) with \(X,Y\sim\operatorname{Unif}(0,1)\), everywhere in the plane. The coordinates are independent: learning \(X\) leaves \(Y\) Uniform on \([0,1]\).
Disk. For the unit disk \(x^2+y^2\le1\), \(f_{X,Y}(x,y)=1/\pi\) inside and 0 outside.
The marginal is not Uniform: values near 0 are more likely because the disk is tallest there.
Incorrect: Inside the disk the joint PDF is the constant \(\frac1\pi=\frac1\pi\cdot1\), a function of \(x\) times a function of \(y\), so \(X\) and \(Y\) are independent.
Correction: Factorization must hold everywhere, including the support. Writing the disk PDF over the whole plane requires the indicator \(I(x^2+y^2\le1)\), which does not split into a function of \(x\) times a function of \(y\). A joint PDF that factors on a rectangle, like the square, does indicate independence.
Try these before reading the answers.
\(E(X^5)=5!=120\). \(E\left((X/3)^3\right)=3!/27=2/9\).
\((1-t)^{-2}=\sum_{n\ge0}(n+1)t^n\), so \(E(X^n)=(n+1)\,n!\): \(E(X)=2\), \(E(X^2)=6\), \(\operatorname{Var}(X)=2\). It is the MGF of a sum of two independent \(\operatorname{Expo}(1)\) variables, since \((1-t)^{-1}(1-t)^{-1}=(1-t)^{-2}\).
\(E(Z^8)=7\cdot5\cdot3\cdot1=105\). Since \(X-\mu=\sigma Z\), \(E\left((X-\mu)^4\right)=\sigma^4E(Z^4)=3\sigma^4\).
The sum is \(\operatorname{Pois}(10)\), so \(P(\text{sum}=0)=e^{-10}\approx4.54\times10^{-5}\).
The sum is 0 only if all three are 0. By independence, \(e^{-2}e^{-3}e^{-5}=e^{-10}\).
The triangle has area \(1/2\), so \(f(x,y)=2\) on \(0<y<x<1\). Then \(f_X(x)=\int_0^x2\,dy=2x\) on \((0,1)\) and \(f_Y(y)=\int_y^12\,dx=2(1-y)\) on \((0,1)\). Given \(X=x\), \(f_{Y\mid X}(y\mid x)=2/(2x)=1/x\) on \((0,x)\), so \(Y\mid X=x\sim\operatorname{Unif}(0,x)\). They are dependent: the conditional range depends on \(x\), and \(f(0.2,0.8)=0\ne f_X(0.2)f_Y(0.8)\).
The region \(x>0.5\) inside the unit disk has area \(\cos^{-1}(0.5)-0.5\sqrt{1-0.25}=\frac\pi3-\frac{\sqrt3}4\approx0.614\). Dividing by \(\pi\) gives \(P(X>0.5)\approx0.196\).
\(\int_0^1\int_0^1xy\,dx\,dy=\frac14\), so \(c=4\). Then \(f(x,y)=(2x)(2y)\) on a square, so \(X\) and \(Y\) are independent with marginal PDFs \(2x\) and \(2y\).
\(P(Y\ge1\mid X=0)=\frac{0.20+0.10}{0.40}=0.75\). \(P(X=1\mid Y=2)=\frac{0.10}{0.20}=0.5\).
| Concept | Essential fact |
|---|---|
| “Moment” | Mean = center of mass; variance = moment of inertia |
| Moments from series | \(E(X^n)=n!\times\) coefficient of \(t^n\) in \(M(t)\) |
| \(\operatorname{Expo}(1)\) MGF | \(1/(1-t)\), \(t<1\) |
| \(\operatorname{Expo}(\lambda)\) moments | \(E(X^n)=n!/\lambda^n\) |
| Normal moments | Odd 0; \(E(Z^{2k})=(2k-1)!!\); \(E(Z^4)=3\) |
| Poisson MGF | \(e^{\lambda(e^t-1)}\) |
| Poisson sums | Independent \(\operatorname{Pois}(\lambda)+\operatorname{Pois}(\mu)=\operatorname{Pois}(\lambda+\mu)\) |
| Joint PMF/PDF | Describes \((X,Y)\) together; sums or integrates to 1 |
| Marginal | Sum or integrate out the other variable |
| Conditional | Joint divided by marginal of the conditioning variable |
| Reconstruction | Joint = conditional × marginal |
| Independence | Joint = product of marginals everywhere |
| Marginals alone | Cannot detect dependence |
| Uniform on square | Independent coordinates |
| Uniform on disk | Dependent; marginal \(\frac2\pi\sqrt{1-x^2}\) |
Viewing probability as mass: the mean is the center of mass and the variance is the moment of inertia about it.
\((2k-1)!!=(2k-1)(2k-3)\cdots3\cdot1\), the even moments of the standard Normal.
The distribution of several random variables considered together, given by a joint CDF, joint PMF, or joint PDF.
The distribution of one random variable on its own, obtained from the joint distribution by summing or integrating over the other variables.
The distribution of one variable given the value of another: the corresponding slice of the joint, renormalized to total probability 1.
A joint PDF equal to \(1/\text{area}(R)\) on a region \(R\) and 0 elsewhere, so that probability is proportional to area.
Random variables are independent exactly when their joint PMF or PDF equals the product of the marginals, or of some function of \(x\) and some function of \(y\), for all points in the plane.
A joint PDF is the mixed partial derivative of the joint CDF, and probabilities of regions are double integrals of it. LOTUS extends to two variables: \(E(g(X,Y))\) integrates \(g\) against the joint PDF without finding the distribution of \(g(X,Y)\). One consequence is that independent random variables satisfy \(E(XY)=E(X)E(Y)\). Two-dimensional LOTUS also gives the expected distance between two independent Uniforms, \(E\lvert X-Y\rvert=1/3\), which can be cross-checked through the maximum and minimum. Finally, when a Poisson number of eggs each hatch independently, the hatched and unhatched counts turn out to be independent Poissons, a fact special to the Poisson.
| Symbol | Meaning |
|---|---|
| \(F(x,y)=P(X\le x,Y\le y)\) | Joint CDF |
| \(f(x,y)=\dfrac{\partial^2F}{\partial x\,\partial y}\) | Joint PDF |
| \(f_X\), \(f_Y\) | Marginal PDFs |
| \(f_{Y\mid X}(y\mid x)\) | Conditional PDF of \(Y\) given \(X=x\) |
| \(g(X,Y)\) | A function of two random variables |
| \(M=\max(X,Y)\), \(L=\min(X,Y)\) | Larger and smaller of \(X\) and \(Y\) |
| \(N\) | Number of eggs |
| \(p\), \(q=1-p\) | Hatching and non-hatching probabilities |
| \(X\), \(Y=N-X\) | Numbers of hatched and unhatched eggs |
After studying these notes, you should be able to:
Lecture 18 introduced joint, marginal, and conditional distributions. The continuous case is collected here.
Setting integration limits. For a region that is not a rectangle, fix the outer variable first and describe the range of the inner variable as a function of it. The Uniform point in the unit disk from Lecture 18 is an example: for fixed \(x\), \(y\) runs from \(-\sqrt{1-x^2}\) to \(\sqrt{1-x^2}\). Comparing its conditional distribution, Uniform on that interval, with its marginal, \(\frac2\pi\sqrt{1-y^2}\), shows directly that the coordinates are dependent.
Let \(X,Y\) be i.i.d. \(\operatorname{Expo}(1)\), so \(f(x,y)=e^{-x}e^{-y}\) for \(x,y>0\). Find \(P(X<Y)\) and \(P(X+Y<1)\).
\(P(X<Y)\). The region is above the diagonal. Fix \(y\) outside, let \(x\) run from 0 to \(y\):
\[P(X<Y)=\int_0^\infty\int_0^ye^{-x}e^{-y}\,dx\,dy=\int_0^\infty(1-e^{-y})e^{-y}\,dy=1-\frac12=\frac12,\]as symmetry predicts.
\(P(X+Y<1)\). Fix \(x\in(0,1)\); then \(y\) runs from 0 to \(1-x\):
\[\int_0^1\int_0^{1-x}e^{-x}e^{-y}\,dy\,dx=\int_0^1e^{-x}\left(1-e^{-(1-x)}\right)dx=(1-e^{-1})-e^{-1}=1-2e^{-1}\approx0.264.\]For a function \(g:\mathbb R^2\to\mathbb R\),
\[E\big(g(X,Y)\big)=\iint g(x,y)f(x,y)\,dx\,dy\quad\text{(continuous)},\qquad E\big(g(X,Y)\big)=\sum_x\sum_yg(x,y)P(X=x,Y=y)\quad\text{(discrete)}.\]As in one dimension, the probabilities stay those of the pair \((X,Y)\); only the values change from \((x,y)\) to \(g(x,y)\). There is no need to find the distribution of \(g(X,Y)\) first.
If \(X\) and \(Y\) are independent with finite means, then \(E(XY)=E(X)E(Y)\).
Proof (continuous case). Apply 2D LOTUS with \(g(x,y)=xy\), factor the joint PDF, and separate the double integral:
\[E(XY)=\iint xy\,f_X(x)f_Y(y)\,dx\,dy=\left(\int xf_X(x)\,dx\right)\left(\int yf_Y(y)\,dy\right)=E(X)E(Y).\]The discrete case is identical with sums. \(\blacksquare\)
More generally, \(E\big(g(X)h(Y)\big)=E\big(g(X)\big)E\big(h(Y)\big)\) for independent \(X,Y\). This justifies the MGF product rule used in Lectures 17 and 18: \(E(e^{tX}e^{tY})=E(e^{tX})E(e^{tY})\).
Random variables with \(E(XY)=E(X)E(Y)\) are called uncorrelated. Independence implies uncorrelated, but not conversely.
Incorrect: If \(E(XY)=E(X)E(Y)\), then \(X\) and \(Y\) are independent.
Correction: Let \(X\) be uniform on \(\{-1,0,1\}\) and \(Y=X^2\). Then \(E(XY)=E(X^3)=0=E(X)E(Y)\), yet \(Y\) is a function of \(X\). A product condition on means captures only one kind of linear association, not all dependence.
Let \(X,Y\) be i.i.d. \(\operatorname{Unif}(0,1)\), two random points on a unit stick. Find \(E\lvert X-Y\rvert\).
By 2D LOTUS. The joint PDF is 1 on the unit square. Split at the diagonal, where \(\lvert x-y\rvert\) changes formula, and use symmetry between the two triangles:
\[E\lvert X-Y\rvert=\int_0^1\int_0^1\lvert x-y\rvert\,dx\,dy=2\int_0^1\int_y^1(x-y)\,dx\,dy.\]The inner integral is \(\int_y^1(x-y)\,dx=\frac{(1-y)^2}2\). Then
\[2\int_0^1\frac{(1-y)^2}2\,dy=\int_0^1(1-y)^2\,dy=\frac13.\]By the maximum and minimum. Let \(M=\max(X,Y)\) and \(L=\min(X,Y)\). Then:
The two points split the stick into three pieces with expected lengths \(E(L)=\frac13\), \(E(M-L)=\frac13\), and \(E(1-M)=\frac13\). By symmetry each piece has the same expected length.
Incorrect: \(E\lvert X-Y\rvert=\lvert E(X)-E(Y)\rvert=0\).
Correction: The absolute value is nonlinear, and \(\lvert X-Y\rvert\) is positive with probability 1, so its mean cannot be 0. Expectation can be moved inside only linear functions.
A hen lays \(N\sim\operatorname{Pois}(\lambda)\) eggs. Each egg hatches with probability \(p\), independently of the other eggs and of \(N\). Let \(X\) be the number that hatch and \(Y=N-X\) the number that do not. Find the joint distribution of \(X\) and \(Y\).
Structure. Conditional on \(N=n\), \(X\mid N=n\sim\operatorname{Bin}(n,p)\). Since \(N=X+Y\), the event \(\{X=i,Y=j\}\) is the same as \(\{N=i+j,X=i\}\).
Joint PMF. Multiply the marginal of \(N\) by the conditional of \(X\):
\[P(X=i,Y=j)=P(N=i+j)\,P(X=i\mid N=i+j)=\frac{e^{-\lambda}\lambda^{i+j}}{(i+j)!}\cdot\frac{(i+j)!}{i!\,j!}p^iq^j.\]The factorials \((i+j)!\) cancel. Write \(e^{-\lambda}=e^{-\lambda p}e^{-\lambda q}\) and \(\lambda^{i+j}p^iq^j=(\lambda p)^i(\lambda q)^j\):
The joint PMF is the product of a \(\operatorname{Pois}(\lambda p)\) PMF in \(i\) and a \(\operatorname{Pois}(\lambda q)\) PMF in \(j\), on the full product of their supports. Therefore:
Why this is surprising. \(X\) and \(Y\) are built from the same total \(N\), so one might expect a large hatched count to signal a large total and hence a large unhatched count. That reasoning would hold for a fixed or non-Poisson total. With a Poisson total, the information about \(N\) carried by \(X\) exactly cancels in its effect on \(Y\). Conversely, if \(N=n\) were fixed, then \(Y=n-X\) would be completely determined by \(X\).
A shop receives \(N\sim\operatorname{Pois}(10)\) customers in an hour, and each independently makes a purchase with probability \(0.3\).
Incorrect: Since \(X\) and \(Y\) add to \(N\), they must be negatively dependent.
Correction: That holds when \(N\) is fixed. When \(N\) is Poisson, the joint PMF factors, so \(X\) and \(Y\) are independent. The conclusion depends on the distribution of the total, not only on the sum relation.
Try these before reading the answers.
\(E(XY)=E(X)E(Y)=\frac14\). \(E\left((X-Y)^2\right)=E(X^2)-2E(X)E(Y)+E(Y^2)=\frac13-\frac12+\frac13=\frac16\).
\(E(XY)=\int_0^1\int_0^1xy(x+y)\,dx\,dy=2\int_0^1\int_0^1x^2y\,dx\,dy=2\cdot\frac13\cdot\frac12=\frac13\). From \(f_X(x)=x+\tfrac12\), \(E(X)=\frac13+\frac14=\frac7{12}=E(Y)\), so \(E(X)E(Y)=\frac{49}{144}\approx0.340\ne\frac13\). They are dependent, consistent with Lecture 18.
\(E(X^2+Y^2)=\frac1\pi\int_0^{2\pi}\int_0^1r^2\cdot r\,dr\,d\theta=\frac1\pi\cdot2\pi\cdot\frac14=\frac12\). The disk is symmetric under \(x\mapsto-x\), so \(E(XY)=0=E(X)E(Y)\). Yet \(X\) and \(Y\) are dependent: another uncorrelated but dependent pair.
\(P(L>l)=P(X>l)P(Y>l)=(1-l)^2\), so \(f_L(l)=2(1-l)\) on \((0,1)\) and \(E(L)=\int_0^12l(1-l)\,dl=1-\frac23=\frac13\).
\(P(\max\le m)=m^3\), so the PDF is \(3m^2\) and the mean is \(\int_0^13m^3\,dm=\frac34\).
\(X\sim\operatorname{Pois}(3)\) and \(Y\sim\operatorname{Pois}(9)\) independently, so \(P(X=0,Y=0)=e^{-3}e^{-9}=e^{-12}\), which is also \(P(N=0)\). Given \(N=8\), \(X\sim\operatorname{Bin}(8,1/4)\): \(P(X=2\mid N=8)=\binom82(1/4)^2(3/4)^6\approx0.311\).
\(X+Y=n\) always, so \(P(X=n,Y=n)=0\) for \(n\ge1\), while \(P(X=n)P(Y=n)=p^nq^n>0\). The joint does not factor; \(Y\) is determined by \(X\).
\(\lvert X-Y\rvert=M-L\) and \(M+L=X+Y\), so \(E(M)=2-E(L)\). With \(L\sim\operatorname{Expo}(2)\), \(E(L)=\frac12\), so \(E(M)=\frac32\) and \(E\lvert X-Y\rvert=1\). By memorylessness, after the first of the two occurs, the remaining wait for the other is a fresh \(\operatorname{Expo}(1)\), which also gives 1.
Expanding and using \(E(XY)=E(X)E(Y)\) gives the identity. Subtracting \((E(X)+E(Y))^2=E(X)^2+2E(X)E(Y)+E(Y)^2\) leaves \(\operatorname{Var}(X)+\operatorname{Var}(Y)\). This proves the additivity used in Lecture 14.
| Concept | Essential fact |
|---|---|
| Joint PDF | \(\partial^2F/\partial x\,\partial y\); integrate over regions |
| Limits | Fix the outer variable; inner limits depend on it |
| Marginal | \(f_X=\int f(x,y)\,dy\) |
| Conditional | \(f_{Y\mid X}=f/f_X\) |
| Independence | \(f=f_Xf_Y\) everywhere, or \(F=F_XF_Y\) |
| 2D LOTUS | \(E(g(X,Y))=\iint g\,f\,dx\,dy\) |
| Product of independents | \(E(XY)=E(X)E(Y)\); more generally \(E(g(X)h(Y))=E(g(X))E(h(Y))\) |
| Uncorrelated | \(E(XY)=E(X)E(Y)\); does not imply independence |
| Expected distance | \(E\lvert X-Y\rvert=1/3\) for i.i.d. \(\operatorname{Unif}(0,1)\) |
| Max and min of two Uniforms | Means \(2/3\) and \(1/3\) |
| Chicken-egg joint PMF | Product of \(\operatorname{Pois}(\lambda p)\) and \(\operatorname{Pois}(\lambda q)\) PMFs |
| Poisson splitting | Hatched and unhatched counts are independent Poissons |
| Fixed total | Splitting a fixed \(n\) gives dependent counts |
The mixed partial derivative of the joint CDF of continuous random variables; its integral over a region is the probability of that region.
\(E(g(X,Y))=\iint g(x,y)f(x,y)\,dx\,dy\), or the corresponding double sum for discrete variables.
Random variables with \(E(XY)=E(X)E(Y)\). Independent variables are uncorrelated; uncorrelated variables need not be independent.
The largest and smallest of a collection of random variables. For two variables, their difference is the distance and their sum is the total.
If each of \(N\sim\operatorname{Pois}(\lambda)\) items is independently classified as type 1 with probability \(p\), the type counts are independent \(\operatorname{Pois}(\lambda p)\) and \(\operatorname{Pois}(\lambda q)\) variables.
The expected distance between two independent standard Normals needs no double integral: their difference is \(\mathcal N(0,2)\), so \(E\lvert Z_1-Z_2\rvert=\sqrt2\,E\lvert Z\rvert=2/\sqrt\pi\). The Multinomial distribution extends the Binomial from two categories to \(k\). It counts how many of \(n\) independent objects fall in each category. Each count is Binomial, merging categories gives another Multinomial, and conditioning on one count leaves a Multinomial on the rest with renormalized probabilities. The Cauchy distribution is the ratio of two independent standard Normals. Its PDF \(\frac1{\pi(1+t^2)}\) follows from a CDF calculation, and its tails are so heavy that its mean does not exist.
| Symbol | Meaning |
|---|---|
| \(Z_1,Z_2\) | i.i.d. \(\mathcal N(0,1)\) |
| \(\mathbf X=(X_1,\ldots,X_k)\) | Category counts |
| \(\mathbf p=(p_1,\ldots,p_k)\) | Category probabilities, \(p_j\ge0\), \(\sum_jp_j=1\) |
| \(\mathbf X\sim\operatorname{Mult}_k(n,\mathbf p)\) | Multinomial with \(n\) objects and \(k\) categories |
| \(\dfrac{n!}{n_1!\cdots n_k!}\) | Multinomial coefficient |
| \(p_j'=p_j/(1-p_1)\) | Renormalized probability after conditioning on \(X_1\) |
| \(T=X/Y\) | Ratio of i.i.d. \(\mathcal N(0,1)\) variables |
| \(\operatorname{Cauchy}\) | Distribution of \(T\) |
| \(\Phi\), \(\varphi\) | Standard Normal CDF and PDF |
After studying these notes, you should be able to:
Let \(Z_1,Z_2\) be i.i.d. \(\mathcal N(0,1)\). Find \(E\lvert Z_1-Z_2\rvert\).
Two-dimensional LOTUS would require integrating \(\lvert z_1-z_2\rvert\varphi(z_1)\varphi(z_2)\) over the plane. A shorter route uses facts already proved.
For i.i.d. \(\mathcal N(\mu,\sigma^2)\) variables, the mean cancels and the scale multiplies, giving \(2\sigma/\sqrt\pi\).
Core idea: An expectation depends only on the distribution of the quantity inside. Identify that distribution first, and the integral may become one-dimensional or already known.
Each of \(n\) objects is placed independently into one of \(k\) categories, landing in category \(j\) with probability \(p_j\), where \(p_j\ge0\) and \(p_1+\cdots+p_k=1\). Let \(X_j\) be the number of objects in category \(j\). Then
\[\mathbf X=(X_1,\ldots,X_k)\sim\operatorname{Mult}_k(n,\mathbf p).\]The counts are always dependent: they add to \(n\), so the last count is determined by the others. With \(k=2\), \(X_1\sim\operatorname{Bin}(n,p_1)\) and \(X_2=n-X_1\).
Joint PMF. Fix counts \(n_1,\ldots,n_k\) with \(n_1+\cdots+n_k=n\).
and 0 otherwise.
Roll a fair die 12 times and record how many times each face appears. The counts are \(\operatorname{Mult}_6(12,(1/6,\ldots,1/6))\). The probability that each face appears exactly twice is
\[\frac{12!}{(2!)^6}\left(\frac16\right)^{12}=\frac{7{,}484{,}400}{2{,}176{,}782{,}336}\approx0.0034.\]Perfectly balanced outcomes are rare, even though each face is equally likely.
To study category \(j\) alone, call it “success” and every other category “failure”. Each object succeeds independently with probability \(p_j\), so
\[X_j\sim\operatorname{Bin}(n,p_j),\qquad E(X_j)=np_j,\qquad\operatorname{Var}(X_j)=np_j(1-p_j).\]No summation over the joint PMF is needed: the story gives the marginal directly.
Merging categories produces another Multinomial, with the merged probabilities added. For example, if the last \(k-2\) categories are combined,
\[(X_1,X_2,X_3+\cdots+X_k)\sim\operatorname{Mult}_3\big(n,(p_1,p_2,p_3+\cdots+p_k)\big).\]Each object lands in the merged category exactly when it lands in one of its parts, with probability the sum of their probabilities, and objects remain independent.
Given \(X_1=n_1\), the other \(n-n_1\) objects are known not to be in category 1. Each of them is in category \(j\ge2\) with conditional probability
\[p_j'=P(\text{category }j\mid\text{not category }1)=\frac{p_j}{1-p_1}=\frac{p_j}{p_2+\cdots+p_k}.\]These objects are still independent of each other, so
Suppose blood types O, A, B, AB occur with probabilities \(0.45,0.40,0.11,0.04\), and 10 unrelated people are typed. Let \(\mathbf X=(X_O,X_A,X_B,X_{AB})\sim\operatorname{Mult}_4(10,\mathbf p)\).
(a) A specific outcome.
\[P(4,4,1,1)=\frac{10!}{4!\,4!\,1!\,1!}(0.45)^4(0.40)^4(0.11)(0.04)\approx0.0291.\](b) A marginal. \(X_A\sim\operatorname{Bin}(10,0.40)\), so \(P(X_A=4)=\binom{10}4(0.4)^4(0.6)^6\approx0.251\) and \(E(X_A)=4\).
(c) Lumping. Combining B and AB gives \((X_O,X_A,X_B+X_{AB})\sim\operatorname{Mult}_3(10,(0.45,0.40,0.15))\).
(d) Conditioning. Given \(X_O=4\), the other 6 people are typed among A, B, AB with probabilities \(\frac{0.40}{0.55},\frac{0.11}{0.55},\frac{0.04}{0.55}\approx(0.727,0.200,0.073)\). So \(E(X_A\mid X_O=4)=6(0.727)\approx4.36\), compared with \(E(X_A)=4\).
Incorrect: Given \(X_1=n_1\), the remaining counts are \(\operatorname{Mult}_{k-1}(n-n_1,(p_2,\ldots,p_k))\).
Correction: The probabilities \(p_2,\ldots,p_k\) sum to \(1-p_1\), not 1, so they are not a valid probability vector. Each must be divided by \(1-p_1\).
Let \(X,Y\) be i.i.d. \(\mathcal N(0,1)\). The distribution of \(T=X/Y\) is the Cauchy distribution.
The ratio is defined with probability 1, since \(P(Y=0)=0\). The plan is to find the CDF and then differentiate.
The sign of \(Y\) complicates the inequality \(X/Y\le t\). Symmetry removes it: \((X,Y)\) and \((X\cdot\text{sign}(Y),\lvert Y\rvert)\) give the same ratio, and \(X\cdot\text{sign}(Y)\) is still \(\mathcal N(0,1)\) and independent of \(\lvert Y\rvert\). So \(T\) has the same distribution as \(X/\lvert Y\rvert\), and
\[P(T\le t)=P\left(X\le t\lvert Y\rvert\right).\]Condition on \(Y=y\) and use independence: given \(Y=y\), the probability is \(\Phi(t\lvert y\rvert)\). Averaging over \(Y\) (the continuous law of total probability):
\[F(t)=\int_{-\infty}^\infty\Phi(t\lvert y\rvert)\varphi(y)\,dy=2\int_0^\infty\Phi(ty)\varphi(y)\,dy.\]The last step uses that the integrand is even in \(y\). This integral has no elementary antiderivative in \(y\), but its derivative in \(t\) is easy.
Differentiate under the integral sign, using \(\frac{d}{dt}\Phi(ty)=y\varphi(ty)\):
\[f(t)=2\int_0^\infty y\varphi(ty)\varphi(y)\,dy=\frac2{2\pi}\int_0^\infty ye^{-(1+t^2)y^2/2}\,dy.\]Substitute \(u=(1+t^2)y^2/2\), so \(du=(1+t^2)y\,dy\):
\[f(t)=\frac1\pi\cdot\frac1{1+t^2}\int_0^\infty e^{-u}\,du=\frac1{\pi(1+t^2)}.\]The CDF follows by integrating the PDF, since \(\frac{d}{dt}\arctan t=\frac1{1+t^2}\), with the constant \(\frac12\) fixed by \(F(-\infty)=0\). Checks: \(F(0)=\frac12\) by symmetry, and \(F(1)=\frac34\), so the quartiles are \(\pm1\).
By LOTUS,
\[E\lvert T\rvert=\frac2\pi\int_0^\infty\frac{t}{1+t^2}\,dt=\frac1\pi\Big[\ln(1+t^2)\Big]_0^\infty=\infty.\]So \(E(T)\) does not exist, even though the PDF is symmetric about 0. The two half-integrals are \(+\infty\) and \(-\infty\), and \(\infty-\infty\) is undefined. The variance and all higher moments are also infinite or undefined, and the MGF does not exist.
The reason is the tail. \(P(\lvert T\rvert>t)\approx\frac2{\pi t}\) for large \(t\), which decays only like \(1/t\). For the Normal, the tail decays like \(e^{-t^2/2}\).
About one Cauchy draw in 16 exceeds 10 in absolute value. For a standard Normal, \(P(\lvert Z\rvert>10)\approx1.5\times10^{-23}\).
If \(T_1,\ldots,T_n\) are i.i.d. Cauchy, their average is again Cauchy, with the same distribution as a single \(T_j\) (a result proved with tools beyond this lecture). Averaging more Cauchy observations therefore gives no more precision. The occasional enormous value keeps dragging the average away.
Incorrect: The Cauchy PDF is symmetric about 0, so its mean is 0.
Correction: Symmetry gives a mean of 0 only when the mean exists, that is, when \(E\lvert T\rvert<\infty\). For the Cauchy, \(E\lvert T\rvert=\infty\). The median is 0; the mean is undefined.
Try these before reading the answers.
\(X-Y\sim\mathcal N(0,8)\), so \(E\lvert X-Y\rvert=\sqrt8\sqrt{2/\pi}=4/\sqrt\pi\approx2.257\), matching \(2\sigma/\sqrt\pi\) with \(\sigma=2\).
\(Z_1+Z_2+Z_3\sim\mathcal N(0,3)\), so the answer is \(\sqrt3\sqrt{2/\pi}=\sqrt{6/\pi}\approx1.382\).
\(P(4,3,1)=\frac{8!}{4!\,3!\,1!}(0.5)^4(0.3)^3(0.2)=280(0.0625)(0.027)(0.2)\approx0.0945\). The neutral count is \(\operatorname{Bin}(8,0.3)\) with mean \(2.4\).
“Not agree” merges two categories with total probability \(0.5\), so the count is \(\operatorname{Bin}(8,0.5)\).
The other 3 responses are neutral or disagree with probabilities \(0.3/0.5=0.6\) and \(0.4\). So the neutral count given 5 agrees is \(\operatorname{Bin}(3,0.6)\).
With \(p_j=1/k\), the PMF values are \(\frac{n!}{n_1!\cdots n_k!}k^{-n}\) and must sum to 1. Multiplying by \(k^n\) gives the identity. Directly: each of \(n\) objects has \(k\) choices.
\(F(\sqrt3)-F(-1)=\frac1\pi\left(\arctan\sqrt3-\arctan(-1)\right)=\frac1\pi\left(\frac\pi3+\frac\pi4\right)=\frac7{12}\).
\(1/T=Y/X\), the ratio of the same two i.i.d. Normals in the other order, which has the same distribution as \(X/Y\).
Solving \(u=\frac12+\frac1\pi\arctan t\) gives \(t=\tan\left(\pi(u-\tfrac12)\right)=F^{-1}(u)\). By universality, \(F^{-1}(U)\) has the Cauchy CDF.
| Concept | Essential fact |
|---|---|
| Difference of i.i.d. \(\mathcal N(0,1)\) | \(Z_1-Z_2\sim\mathcal N(0,2)\) |
| \(E\lvert Z\rvert\) | \(\sqrt{2/\pi}\) |
| Expected Normal distance | \(2/\sqrt\pi\), or \(2\sigma/\sqrt\pi\) |
| Multinomial story | \(n\) independent objects into \(k\) categories |
| Multinomial PMF | \(\frac{n!}{n_1!\cdots n_k!}p_1^{n_1}\cdots p_k^{n_k}\) |
| Marginal | \(X_j\sim\operatorname{Bin}(n,p_j)\) |
| Lumping | Merge categories; add their probabilities |
| Conditioning | Remaining counts Multinomial with \(p_j/(1-p_1)\) |
| Cauchy definition | \(X/Y\) for i.i.d. \(\mathcal N(0,1)\) |
| Cauchy PDF | \(\frac1{\pi(1+t^2)}\) |
| Cauchy CDF | \(\frac12+\frac1\pi\arctan t\) |
| Cauchy mean | Does not exist; median 0 |
| Tail | \(P(\lvert T\rvert>t)\approx2/(\pi t)\) |
The joint distribution of category counts when \(n\) objects are independently assigned to \(k\) categories with fixed probabilities.
\(\frac{n!}{n_1!\cdots n_k!}\), the number of ways to divide \(n\) labeled objects into groups of sizes \(n_1,\ldots,n_k\).
Merging Multinomial categories yields a Multinomial whose probabilities are sums of the merged probabilities.
The distribution of the ratio of two i.i.d. standard Normals, with PDF \(\frac1{\pi(1+t^2)}\). It has no mean.
A tail probability that decays slowly, such as like a power of \(t\) rather than exponentially, so that extreme values remain non-negligible and moments may fail to exist.
Variance is not linear, so the variance of a sum needs a new tool: covariance, which measures how two random variables vary together. Covariance is symmetric, ignores added constants, and is bilinear, so it expands like a product of sums. With it, \(\operatorname{Var}(X+Y)=\operatorname{Var}(X)+\operatorname{Var}(Y)+2\operatorname{Cov}(X,Y)\). Independent random variables are uncorrelated, but uncorrelated ones need not be independent. Correlation is covariance after standardizing, so it has no units and always lies between \(-1\) and \(1\). These tools give the negative covariance of two Multinomial counts, the Binomial variance in one line, and the first step of the Hypergeometric variance.
| Symbol | Meaning |
|---|---|
| \(\operatorname{Cov}(X,Y)\) | Covariance, \(E\big((X-EX)(Y-EY)\big)\) |
| \(\operatorname{Corr}(X,Y)\), \(\rho\) | Correlation, the covariance of the standardized variables |
| \(\operatorname{SD}(X)\) | Standard deviation, \(\sqrt{\operatorname{Var}(X)}\) |
| \(\dfrac{X-EX}{\operatorname{SD}(X)}\) | Standardized \(X\): mean 0, variance 1 |
| \(I_A\) | Indicator random variable of event \(A\) |
| \(\mathbf X\sim\operatorname{Mult}_k(n,\mathbf p)\) | Multinomial category counts |
| \(X\sim\operatorname{HGeom}(w,b,n)\) | White balls in a sample of \(n\) drawn without replacement from \(w\) white and \(b\) black |
After studying these notes, you should be able to:
For random variables \(X\) and \(Y\) on the same sample space,
\[\boxed{\operatorname{Cov}(X,Y)=E\big((X-EX)(Y-EY)\big).}\]The product brings the two variables together. If \(X\) tends to be above its mean when \(Y\) is above its mean, and below when \(Y\) is below, the two deviations usually have the same sign, so the product is usually positive and the covariance is positive: \(X\) and \(Y\) are positively correlated. If being above the mean in one tends to go with being below the mean in the other, the product is usually negative, and they are negatively correlated.
Expanding the product and using linearity gives the formula used most often in practice:
\[\begin{aligned} \operatorname{Cov}(X,Y) &=E\big(XY-X\,EY-Y\,EX+EX\,EY\big)\\ &=E(XY)-EX\,EY-EX\,EY+EX\,EY\\ &=E(XY)-E(X)E(Y). \end{aligned}\]The definition is better for intuition; the second form is usually easier to compute. With \(Y=X\), it becomes \(E(X^2)-(EX)^2\), the familiar variance formula.
Each property follows by substituting into the definition, or into \(E(XY)-E(X)E(Y)\), and using linearity of expectation.
| Property | Statement |
|---|---|
| 1. Variance | \(\operatorname{Cov}(X,X)=\operatorname{Var}(X)\) |
| 2. Symmetry | \(\operatorname{Cov}(X,Y)=\operatorname{Cov}(Y,X)\) |
| 3. Constants | \(\operatorname{Cov}(X,c)=0\) for any constant \(c\) |
| 4. Scaling | \(\operatorname{Cov}(cX,Y)=c\operatorname{Cov}(X,Y)\) |
| 5. Sums | \(\operatorname{Cov}(X,Y+Z)=\operatorname{Cov}(X,Y)+\operatorname{Cov}(X,Z)\) |
For property 3, \(c-E(c)=0\), so the product inside the expectation is 0. For property 5, replace \(Y\) by \(Y+Z\): \(E(X(Y+Z))=E(XY)+E(XZ)\) and \(E(Y+Z)=EY+EZ\), and the terms regroup into the two covariances.
Properties 4 and 5 together are called bilinearity: holding one argument fixed, covariance is linear in the other. Adding a constant changes nothing either, since \(\operatorname{Cov}(X+c,Y)=\operatorname{Cov}(X,Y)+\operatorname{Cov}(c,Y)=\operatorname{Cov}(X,Y)\).
Bilinearity works like the distributive law. For example,
\[\operatorname{Cov}(X+Y,Z+W)=\operatorname{Cov}(X,Z)+\operatorname{Cov}(X,W)+\operatorname{Cov}(Y,Z)+\operatorname{Cov}(Y,W).\]In general, for constants \(a_i,b_j\),
Every term on the left is paired with every term on the right, and constants come out. Using these rules directly is usually much easier than returning to the definition.
By property 1 and bilinearity,
\[\operatorname{Var}(X_1+X_2)=\operatorname{Cov}(X_1+X_2,X_1+X_2) =\operatorname{Var}(X_1)+\operatorname{Var}(X_2)+\operatorname{Cov}(X_1,X_2)+\operatorname{Cov}(X_2,X_1).\]The two cross terms are equal by symmetry, so
For \(n\) terms,
\[\boxed{\operatorname{Var}\left(\sum_{j=1}^n X_j\right)=\sum_{j=1}^n\operatorname{Var}(X_j)+2\sum_{i<j}\operatorname{Cov}(X_i,X_j).}\]The factor 2 is there because the sum runs over \(i<j\), so each pair appears once although \(\operatorname{Cov}(X_i,X_j)\) and \(\operatorname{Cov}(X_j,X_i)\) both occur. Summing over all \(i\ne j\) instead would need no factor 2.
The variance of a sum equals the sum of the variances exactly when the covariance terms add to zero. Independence is one way for this to happen, but not the only way.
For a difference, write \(X-Y=X+(-Y)\):
\[\operatorname{Var}(X-Y)=\operatorname{Var}(X)+\operatorname{Var}(Y)-2\operatorname{Cov}(X,Y).\]The variances still add; only the covariance term changes sign.
Roll two fair dice independently. Let \(X\) be the first die and \(S=X+Y\) the total, where \(Y\) is the second die. Each die has variance \(35/12\).
By bilinearity and independence of \(X\) and \(Y\),
\[\operatorname{Cov}(X,S)=\operatorname{Cov}(X,X)+\operatorname{Cov}(X,Y)=\frac{35}{12}+0=\frac{35}{12},\] \[\operatorname{Var}(S)=\frac{35}{12}+\frac{35}{12}=\frac{35}{6}.\]The covariance is positive: a high first die tends to make the total high. Correlation (Part 5) puts this on a fixed scale:
\[\operatorname{Corr}(X,S)=\frac{35/12}{\sqrt{(35/12)(35/6)}}=\frac1{\sqrt2}\approx0.707.\]If \(X\) and \(Y\) are independent, then \(\operatorname{Cov}(X,Y)=0\). Random variables with zero covariance are called uncorrelated.
Proof. Independence gives \(E(XY)=E(X)E(Y)\), shown in Lecture 19 with two-dimensional LOTUS. So \(\operatorname{Cov}(X,Y)=E(XY)-E(X)E(Y)=0\).
The converse is false.
Let \(Z\sim\mathcal N(0,1)\), \(X=Z\), and \(Y=Z^2\). The odd moments of a standard Normal are 0, so
\[\operatorname{Cov}(X,Y)=E(Z^3)-E(Z)E(Z^2)=0-0\cdot1=0.\]So \(X\) and \(Y\) are uncorrelated. They are extremely dependent: \(Y\) is a function of \(X\), so knowing \(X\) determines \(Y\) completely. Knowing \(Y\) also gives information about \(X\): it determines \(\lvert X\rvert=\sqrt Y\), leaving only the sign unknown.
A stronger statement is true but beyond these notes: if every function of \(X\) is uncorrelated with every function of \(Y\), then \(X\) and \(Y\) are independent. Zero correlation between \(X\) and \(Y\) themselves is not enough.
Incorrect: The covariance is 0, so \(X\) and \(Y\) are independent.
Correction: Zero covariance rules out a linear trend only. \(X=Z\) and \(Y=Z^2\) are uncorrelated but completely dependent. Independence implies uncorrelated; the reverse direction fails.
Covariance depends on units. If \(X\) and \(Y\) are lengths, measuring them in nanometers instead of light-years changes the covariance by an enormous factor, so a covariance of 42 says little by itself. Correlation removes the units by standardizing first.
The two forms agree: subtracting the means is adding constants, which does not change covariance, and the standard deviations come out as constants by property 4. The standardized variables have mean 0 and variance 1 and are unitless, since a measurement in nanometers minus nanometers, divided by nanometers, has no units.
For any \(X\) and \(Y\) with positive finite variances,
\[\boxed{-1\le\operatorname{Corr}(X,Y)\le1.}\]Proof. Standardizing does not change the correlation, so assume without loss of generality that \(X\) and \(Y\) already have mean 0 and variance 1. Then \(\operatorname{Cov}(X,Y)=\operatorname{Corr}(X,Y)=\rho\), and
\[\operatorname{Var}(X+Y)=1+1+2\rho=2+2\rho,\qquad \operatorname{Var}(X-Y)=1+1-2\rho=2-2\rho.\]Variances are nonnegative, so \(2+2\rho\ge0\) and \(2-2\rho\ge0\), which gives \(-1\le\rho\le1\). \(\blacksquare\)
The proof also shows when the extremes occur: \(\rho=1\) exactly when \(\operatorname{Var}(X-Y)=0\), so the standardized variables are equal with probability 1, and \(\rho=-1\) exactly when they are negatives of each other. A correlation of \(0.9\) is therefore high, because the largest possible value is 1. This inequality is a form of the Cauchy–Schwarz inequality.
Let \(\mathbf X=(X_1,\ldots,X_k)\sim\operatorname{Mult}_k(n,\mathbf p)\). Each count is Binomial, so \(\operatorname{Cov}(X_i,X_i)=\operatorname{Var}(X_i)=np_i(1-p_i)\). For two different categories, the covariance should be negative: the \(n\) objects are fixed in number, so more objects in category 1 leaves fewer for category 2.
Find \(c=\operatorname{Cov}(X_1,X_2)\) by computing \(\operatorname{Var}(X_1+X_2)\) in two ways. By the variance of a sum,
\[\operatorname{Var}(X_1+X_2)=np_1(1-p_1)+np_2(1-p_2)+2c.\]By the lumping property, merging categories 1 and 2 gives a Binomial count with success probability \(p_1+p_2\):
\[\operatorname{Var}(X_1+X_2)=n(p_1+p_2)(1-p_1-p_2).\]Setting these equal and expanding, the terms \(np_1-np_1^2+np_2-np_2^2\) cancel from both sides, leaving \(2c=-2np_1p_2\). Relabeling gives the general result:
The covariance is negative, as predicted.
Roll a fair die 12 times. Let \(X_1\) and \(X_2\) be the numbers of ones and twos. Then
\[\operatorname{Cov}(X_1,X_2)=-12\cdot\frac16\cdot\frac16=-\frac13,\qquad \operatorname{Var}(X_1)=\operatorname{Var}(X_2)=12\cdot\frac16\cdot\frac56=\frac53,\] \[\operatorname{Corr}(X_1,X_2)=\frac{-1/3}{5/3}=-\frac15.\]Two facts about indicator random variables are simple but often overlooked.
Powers of 0 and 1 are 0 and 1, and a product of two 0/1 values is 1 exactly when both are 1.
Bernoulli variance. If \(I\sim\operatorname{Bern}(p)\), then \(I^2=I\), so
\[\operatorname{Var}(I)=E(I^2)-(EI)^2=p-p^2=p(1-p)=pq.\]Binomial variance. Write \(X\sim\operatorname{Bin}(n,p)\) as \(X=I_1+\cdots+I_n\) with \(I_j\) i.i.d. \(\operatorname{Bern}(p)\). The indicators are independent, so every covariance term is 0, and
\[\boxed{\operatorname{Var}(X)=npq.}\]This agrees with the indicator calculation in Lecture 14, now in one line.
Let \(X\sim\operatorname{HGeom}(w,b,n)\): draw \(n\) balls without replacement from an urn with \(w\) white and \(b\) black balls, and count the white ones. Write \(X=I_1+\cdots+I_n\), where \(I_j\) indicates that the \(j\)th ball drawn is white. Now the indicators are dependent, so the covariance terms remain:
\[\operatorname{Var}(X)=\sum_{j=1}^n\operatorname{Var}(I_j)+2\sum_{i<j}\operatorname{Cov}(I_i,I_j).\]Symmetry. Before any ball is drawn, the \(j\)th ball is equally likely to be any of the \(w+b\) balls, so each \(I_j\sim\operatorname{Bern}(p)\) with \(p=w/(w+b)\). These are unconditional statements: no earlier draw is being observed. Likewise every pair of positions has the same joint distribution, so all \(\binom n2\) covariances equal \(\operatorname{Cov}(I_1,I_2)\):
\[\operatorname{Var}(X)=np(1-p)+2\binom n2\operatorname{Cov}(I_1,I_2).\]The covariance. By \(I_1I_2=I_{A_1\cap A_2}\) and the fundamental bridge,
\[\operatorname{Cov}(I_1,I_2)=P(\text{first two balls white})-p^2=\frac{w}{w+b}\cdot\frac{w-1}{w+b-1}-p^2.\]Everything is now known; the remaining step is algebra, completed in Lecture 22.
Incorrect: To find \(P(I_2=1)\), the second ball’s color must be conditioned on the first ball’s color.
Correction: Without observing the first ball, the second ball is equally likely to be any ball in the urn, so \(P(I_2=1)=w/(w+b)\) by symmetry. Conditioning on the first ball is needed only for joint events such as both balls being white.
Try these before reading the answers.
\(\operatorname{Var}(X+Y)=4+9+2(-2)=9\) and \(\operatorname{Var}(X-Y)=4+9-2(-2)=17\).
The constant 3 drops out. Then \(\operatorname{Cov}(2X,Y-X)=2\operatorname{Cov}(X,Y)-2\operatorname{Var}(X)=1-2=-1\).
\(E(X)=0\) and \(E(X^3)=0\) by symmetry, so \(\operatorname{Cov}(X,X^2)=E(X^3)-E(X)E(X^2)=0\). They are dependent: for example, \(P(Y>1/4)=1/2\), but given \(X=0.9\), \(Y=0.81>1/4\) with certainty.
\(\operatorname{Cov}(X,X+Y)=\operatorname{Var}(X)+\operatorname{Cov}(X,Y)=\sigma^2\), and \(\operatorname{Var}(X+Y)=2\sigma^2\). So \(\operatorname{Corr}=\sigma^2/(\sigma\cdot\sqrt2\,\sigma)=1/\sqrt2\), matching the dice example.
\(\operatorname{Cov}(aX+b,cY+d)=ac\operatorname{Cov}(X,Y)\) and \(\operatorname{SD}(aX+b)\operatorname{SD}(cY+d)=\lvert a\rvert\lvert c\rvert\operatorname{SD}(X)\operatorname{SD}(Y)\). So the correlation is \(\operatorname{Corr}(X,Y)\) if \(ac>0\) and \(-\operatorname{Corr}(X,Y)\) if \(ac<0\). Changing units does not change correlation.
\(\operatorname{Cov}(X_1,X_2)=-10(0.5)(0.3)=-1.5\). Then \(\operatorname{Var}(X_1)+\operatorname{Var}(X_2)+2\operatorname{Cov}=2.5+2.1-3=1.6\). By lumping, \(X_1+X_2\sim\operatorname{Bin}(10,0.8)\) with variance \(10(0.8)(0.2)=1.6\). They agree.
For standardized \(\tilde X,\tilde Y\) with correlation 1, \(\operatorname{Var}(\tilde X-\tilde Y)=2-2(1)=0\). A variable with variance 0 equals its mean with probability 1, and \(E(\tilde X-\tilde Y)=0\), so \(\tilde X=\tilde Y\) with probability 1. Equivalently, \(Y\) is an increasing linear function of \(X\).
Let \(I_j\) indicate a match at position \(j\), so \(P(I_j=1)=1/n\) and, for \(i\ne j\), \(P(I_i=I_j=1)=\frac1{n(n-1)}\). Then
\[\operatorname{Var}(M)=n\cdot\frac1n\left(1-\frac1n\right)+2\binom n2\left(\frac1{n(n-1)}-\frac1{n^2}\right) =\left(1-\frac1n\right)+\frac1n=1.\]The mean and variance of the number of matches are both 1.
For i.i.d. variables all covariances are 0, giving \(n\operatorname{Var}(X_1)\). If all \(X_j\) equal \(X_1\), every covariance equals \(\operatorname{Var}(X_1)\), and the sum is \(nX_1\) with variance \(n^2\operatorname{Var}(X_1)\).
| Concept | Essential fact |
|---|---|
| Covariance | \(E((X-EX)(Y-EY))=E(XY)-E(X)E(Y)\) |
| Sign | Positive: tend to be above or below their means together |
| Basic properties | \(\operatorname{Cov}(X,X)=\operatorname{Var}(X)\); symmetric; \(\operatorname{Cov}(X,c)=0\) |
| Bilinearity | \(\operatorname{Cov}(\sum a_iX_i,\sum b_jY_j)=\sum\sum a_ib_j\operatorname{Cov}(X_i,Y_j)\) |
| Variance of a sum | \(\sum\operatorname{Var}(X_j)+2\sum_{i<j}\operatorname{Cov}(X_i,X_j)\) |
| Difference | \(\operatorname{Var}(X-Y)=\operatorname{Var}X+\operatorname{Var}Y-2\operatorname{Cov}(X,Y)\) |
| Independence | Implies uncorrelated; not conversely (\(Z\) and \(Z^2\)) |
| Correlation | Covariance of standardized variables; unitless |
| Bounds | \(-1\le\operatorname{Corr}\le1\), from \(\operatorname{Var}(X\pm Y)\ge0\) |
| Multinomial | \(\operatorname{Cov}(X_i,X_j)=-np_ip_j\) for \(i\ne j\) |
| Indicators | \(I_A^k=I_A\), \(I_AI_B=I_{A\cap B}\) |
| Binomial | \(\operatorname{Var}=npq\) by adding independent Bernoulli variances |
| Hypergeometric setup | \(np(1-p)+2\binom n2\operatorname{Cov}(I_1,I_2)\), with dependent indicators |
\(E((X-EX)(Y-EY))\), a measure of how two random variables vary together. Its sign indicates the direction of their linear association.
Covariance is linear in each argument when the other is held fixed, so the covariance of two sums expands into covariances of all pairs of terms.
Having covariance 0. Independent random variables are uncorrelated, but uncorrelated ones may be dependent.
The covariance of the standardized variables, \(\operatorname{Cov}(X,Y)/(\operatorname{SD}(X)\operatorname{SD}(Y))\). It is unitless and lies between \(-1\) and \(1\).
Replacing \(X\) by \((X-EX)/\operatorname{SD}(X)\), which has mean 0, variance 1, and no units.
\(I_AI_B=I_{A\cap B}\) and \(I_A^k=I_A\), which turn expectations of products of indicators into probabilities of intersections.
The Hypergeometric variance is the Binomial variance times a finite population correction, \(\frac{N-n}{N-1}\). LOTUS gives the mean of a transformed random variable; the change-of-variables formula gives its whole distribution. For a smooth, strictly increasing \(g\) and \(Y=g(X)\), the PDF is \(f_Y(y)=f_X(x)\,dx/dy\), written as a function of \(y\). The Log-Normal shows the method. In several dimensions the derivative becomes the absolute value of a Jacobian determinant. A convolution is the distribution of a sum of independent random variables, found by conditioning on one of them. Finally, probability can prove that an object exists: if a random object has a property with positive probability, or if the average score is high, some object must have that property or score.
| Symbol | Meaning |
|---|---|
| \(N=w+b\), \(p=w/N\) | Population size and fraction of white balls |
| \(\dfrac{N-n}{N-1}\) | Finite population correction |
| \(Y=g(X)\), \(y=g(x)\) | A transformation and its counterpart for values |
| \(\dfrac{dx}{dy}\) | Derivative of \(x=g^{-1}(y)\) with respect to \(y\) |
| \(\dfrac{\partial\mathbf x}{\partial\mathbf y}\) | Jacobian matrix of partial derivatives \(\partial x_i/\partial y_j\) |
| \(\left\lvert\det\dfrac{\partial\mathbf x}{\partial\mathbf y}\right\rvert\) | Absolute value of the Jacobian determinant |
| \(T=X+Y\) | A sum whose distribution is a convolution |
After studying these notes, you should be able to:
Lecture 21 reduced the variance of \(X\sim\operatorname{HGeom}(w,b,n)\) to
\[\operatorname{Var}(X)=np(1-p)+2\binom n2\operatorname{Cov}(I_1,I_2),\qquad \operatorname{Cov}(I_1,I_2)=p\cdot\frac{w-1}{N-1}-p^2,\]where \(N=w+b\) is the population size and \(p=w/N\). Simplify the covariance:
\[\operatorname{Cov}(I_1,I_2)=p\left(\frac{w-1}{N-1}-\frac wN\right) =p\cdot\frac{N(w-1)-w(N-1)}{N(N-1)} =p\cdot\frac{w-N}{N(N-1)} =-\frac{p(1-p)}{N-1}.\]It is negative: drawing a white ball first leaves fewer white balls for the second draw. Then
\[\operatorname{Var}(X)=np(1-p)-n(n-1)\frac{p(1-p)}{N-1} =np(1-p)\left(1-\frac{n-1}{N-1}\right).\]This is the Binomial variance \(np(1-p)\) times a factor called the finite population correction. Sampling without replacement varies less than sampling with replacement.
Checking extreme cases.
Let \(X\) be the number of aces in a five-card hand: \(X\sim\operatorname{HGeom}(4,48,5)\), with \(N=52\) and \(p=1/13\).
\[np(1-p)=5\cdot\frac1{13}\cdot\frac{12}{13}=\frac{60}{169}\approx0.355,\qquad \frac{N-n}{N-1}=\frac{47}{51}\approx0.922,\] \[\operatorname{Var}(X)=\frac{47}{51}\cdot\frac{60}{169}=\frac{940}{2873}\approx0.327.\]LOTUS gives \(E(g(X))\) without finding the distribution of \(g(X)\). Often the whole distribution is needed.
Let \(X\) be continuous with PDF \(f_X\), and let \(Y=g(X)\), where \(g\) is differentiable and strictly increasing on the support of \(X\). Then
\[\boxed{f_Y(y)=f_X(x)\frac{dx}{dy},}\]where \(y=g(x)\), so \(x=g^{-1}(y)\), and the right side is written as a function of \(y\).
Reading the formula. The right side contains \(x\). Substitute \(x=g^{-1}(y)\) so that the answer is a function of \(y\). By the chain rule, \(\frac{dx}{dy}=1\Big/\frac{dy}{dx}\), so either derivative can be computed, whichever is easier. A mnemonic is \(f_Y(y)\,dy=f_X(x)\,dx\): probability in a tiny interval is preserved.
Proof. Find the CDF and differentiate. Since \(g\) is strictly increasing, the events \(\{g(X)\le y\}\) and \(\{X\le g^{-1}(y)\}\) are the same:
\[F_Y(y)=P(g(X)\le y)=P(X\le g^{-1}(y))=F_X(g^{-1}(y))=F_X(x).\]Differentiating with respect to \(y\) by the chain rule gives \(f_Y(y)=f_X(x)\frac{dx}{dy}\). \(\blacksquare\)
If \(g\) is strictly decreasing instead, the same argument gives \(f_Y(y)=f_X(x)\left\lvert\frac{dx}{dy}\right\rvert\). The absolute value keeps the density positive.
Incorrect: For \(Z\sim\mathcal N(0,1)\) and \(Y=Z^2\), plug \(g(z)=z^2\) into the formula.
Correction: \(z^2\) decreases on the negative half-line and increases on the positive half-line, so it is not strictly increasing. Go back to the CDF instead. On a positive support, such as \(X\sim\operatorname{Expo}(1)\), the function \(x^2\) is strictly increasing and the formula applies.
For \(y>0\), use the CDF directly:
\[F_Y(y)=P(-\sqrt y\le Z\le\sqrt y)=2\Phi(\sqrt y)-1.\]Differentiating, \(f_Y(y)=2\varphi(\sqrt y)\cdot\frac1{2\sqrt y}=\frac1{\sqrt{2\pi y}}e^{-y/2}\) for \(y>0\). This distribution is called the Chi-Square with one degree of freedom.
Let \(Z\sim\mathcal N(0,1)\) and \(Y=e^Z\). The name means that \(\log Y=Z\) is Normal. It does not mean the log of a Normal, which would be undefined for negative values.
The function \(y=e^z\) is strictly increasing and differentiable, so the theorem applies.
Since \(P(Y\le1)=P(Z\le0)=\frac12\), the median is 1. The mean, found from the Normal MGF, is \(E(Y)=E(e^Z)=e^{1/2}\approx1.65\), larger than the median because of the long right tail.
Now let \(\mathbf X=(X_1,\ldots,X_n)\) be a continuous random vector with joint PDF \(f_{\mathbf X}\), and \(\mathbf Y=g(\mathbf X)\), where \(g\) maps \(\mathbb R^n\) to \(\mathbb R^n\), is invertible, and has continuous partial derivatives. The analogous formula holds:
again written as a function of \(\mathbf y\). The Jacobian matrix collects every partial derivative of the \(x\)’s with respect to the \(y\)’s:
\[\frac{\partial\mathbf x}{\partial\mathbf y}= \begin{pmatrix} \dfrac{\partial x_1}{\partial y_1}&\cdots&\dfrac{\partial x_1}{\partial y_n}\\ \vdots&\ddots&\vdots\\ \dfrac{\partial x_n}{\partial y_1}&\cdots&\dfrac{\partial x_n}{\partial y_n} \end{pmatrix}.\]The bars mean the absolute value of the determinant: the determinant compresses the matrix to one number, and the absolute value keeps the density positive. As in one dimension, \(\left\lvert\det\frac{\partial\mathbf x}{\partial\mathbf y}\right\rvert=1\Big/\left\lvert\det\frac{\partial\mathbf y}{\partial\mathbf x}\right\rvert\), so compute whichever direction is easier. Writing \(\partial\mathbf x/\partial\mathbf y\) rather than a bare \(J\) makes the direction clear. The proof is an exercise in multivariable calculus.
Let \(X,Y\) be i.i.d. \(\mathcal N(0,1)\), \(U=X+Y\), and \(V=X-Y\). Invert: \(x=\frac{u+v}2\), \(y=\frac{u-v}2\). The Jacobian determinant is
\[\det\begin{pmatrix}1/2&1/2\\1/2&-1/2\end{pmatrix}=-\frac12,\qquad\text{absolute value }\frac12.\]Since \(x^2+y^2=\frac{u^2+v^2}{2}\),
\[f_{U,V}(u,v)=\frac1{2\pi}e^{-(u^2+v^2)/4}\cdot\frac12 =\left(\frac1{\sqrt{4\pi}}e^{-u^2/4}\right)\left(\frac1{\sqrt{4\pi}}e^{-v^2/4}\right).\]The joint PDF factors into two \(\mathcal N(0,2)\) densities, so \(U\) and \(V\) are independent \(\mathcal N(0,2)\).
A convolution is the distribution of a sum of independent random variables. Stories (the Binomial) and MGFs (the Poisson and Normal) have handled some sums already. When neither helps, compute directly. Let \(T=X+Y\) with \(X\) and \(Y\) independent.
Discrete case. For \(T=t\), \(X\) must take some value \(x\) and \(Y\) must make up the rest. These are disjoint cases, and independence splits each probability:
Continuous case.
\[\boxed{f_T(t)=\int_{-\infty}^{\infty}f_X(x)f_Y(t-x)\,dx.}\]The continuous formula looks like the discrete one with PMFs replaced by PDFs, but an analogy is not a proof. Justification. Condition on \(X\) with the continuous law of total probability, then use independence to drop the condition:
\[F_T(t)=\int_{-\infty}^{\infty}P(X+Y\le t\mid X=x)f_X(x)\,dx=\int_{-\infty}^{\infty}F_Y(t-x)f_X(x)\,dx.\]Differentiating with respect to \(t\), and exchanging the derivative and the integral, gives the formula, since \(F_Y'=f_Y\).
Incorrect: By analogy with sums, the PDF of a product \(XY\) is \(\int f_X(x)f_Y(t/x)\,dx\).
Correction: Reasoning by analogy fails here. Conditioning on \(X=x\) and differentiating \(P(Y\le t/x)\) (for \(x>0\)) produces an extra factor \(1/\lvert x\rvert\), so the correct formula is \(\int f_X(x)f_Y(t/x)\frac1{\lvert x\rvert}\,dx\). Derive such formulas through the CDF.
Let \(X,Y\) be i.i.d. \(\operatorname{Expo}(\lambda)\). For \(t>0\), the integrand is nonzero only when \(0<x<t\):
\[f_T(t)=\int_0^t\lambda e^{-\lambda x}\,\lambda e^{-\lambda(t-x)}\,dx=\lambda^2e^{-\lambda t}\int_0^t dx=\lambda^2te^{-\lambda t}.\]When possible, prefer a story or an MGF to a convolution integral; use the integral when nothing else works.
Probability can prove that an object with a desired property exists, even when the problem mentions no randomness.
Strategy 1. Choose an object at random, for example uniformly from a finite collection. Let \(A\) be the event that the chosen object has the property. If \(P(A)>0\), then some object has the property, since otherwise \(P(A)\) would be 0. Only a lower bound on \(P(A)\) is needed, not its exact value.
Strategy 2. Suppose each object has a numerical score. Choose an object at random and let \(X\) be its score. Then
Not every object can be below average. If \(E(X)\) is good, a good object exists, even if none can be exhibited.
Shannon’s theorem. This method underlies information theory. In 1948 Claude Shannon showed that a noisy channel has a capacity, and that messages can be sent at any rate below it with an arbitrarily small probability of error. He proved that good codes exist by showing that a randomly chosen code performs well on average. Explicit codes achieving this were found only decades later. Modern communication, including cell phones, rests on this work.
There are 100 people and 15 committees of 20 people each. Each person is on exactly 3 committees, consistent with \(15\cdot20=300=100\cdot3\). Show that some two committees share at least 3 members.
Searching every pair of committees directly would be hopeless in general. Instead, choose 2 of the 15 committees at random, all \(\binom{15}2\) pairs equally likely, and find the expected overlap.
Some pair of committees has overlap at least \(20/7\). An overlap is an integer, so that pair shares at least 3 members.
Try these before reading the answers.
\(N=10\), \(p=0.6\), and \(np(1-p)=1.2\). The correction is \(\frac{10-5}{10-1}=\frac59\), so \(\operatorname{Var}(X)=\frac59(1.2)=\frac23\). With replacement the variance would be \(1.2\).
\(y=x^3\) is increasing on \((0,1)\), with \(x=y^{1/3}\) and \(\frac{dx}{dy}=\frac13y^{-2/3}\). So \(f_Y(y)=\frac13y^{-2/3}\) for \(0<y<1\).
\(y=-\log u\) is decreasing, with \(u=e^{-y}\) and \(\left\lvert\frac{du}{dy}\right\rvert=e^{-y}\). So \(f_Y(y)=1\cdot e^{-y}\) for \(y>0\): \(Y\sim\operatorname{Expo}(1)\), matching universality of the Uniform.
\(z=\frac{\log y-\mu}\sigma\) and \(\frac{dz}{dy}=\frac1{\sigma y}\). So
\[f_Y(y)=\frac1{\sigma y\sqrt{2\pi}}\exp\left(-\frac{(\log y-\mu)^2}{2\sigma^2}\right),\qquad y>0.\]On \(x>0\), \(y=x^2\) is increasing, \(x=\sqrt y\), and \(\frac{dx}{dy}=\frac1{2\sqrt y}\). So \(f_Y(y)=\frac{e^{-\sqrt y}}{2\sqrt y}\) for \(y>0\).
\(f_T(t)=\int_0^1f_Y(t-x)\,dx\), where \(f_Y(t-x)=1\) when \(0<t-x<1\). For \(0<t\le1\), \(x\) ranges over \((0,t)\), giving \(t\). For \(1<t<2\), \(x\) ranges over \((t-1,1)\), giving \(2-t\). The density is a triangle peaking at \(t=1\).
by the binomial theorem. This confirms the MGF result from Lecture 18.
\(10\cdot9=90=30\cdot3\), so the counts are consistent. For a random pair of committees, \(P(\text{person on both})=\binom32\big/\binom{10}2=\frac3{45}\), so the expected overlap is \(30\cdot\frac3{45}=2\). Some pair shares at least 2 members.
If no object had the property, the probability that a random object has it would be exactly 0. A positive probability, or an average that some object must reach, rules this out, so an object exists without being identified.
| Concept | Essential fact |
|---|---|
| Hypergeometric covariance | \(\operatorname{Cov}(I_1,I_2)=-p(1-p)/(N-1)\) |
| Hypergeometric variance | \(\frac{N-n}{N-1}np(1-p)\) |
| Finite population correction | 1 when \(n=1\); near 1 when \(N\gg n\); 0 when \(n=N\) |
| One-dimensional change of variables | \(f_Y(y)=f_X(x)\lvert dx/dy\rvert\) for strictly monotone smooth \(g\) |
| Check assumptions | Not one-to-one, such as \(Z^2\): return to the CDF |
| Log-Normal | \(Y=e^Z\): \(f_Y(y)=\frac1{y\sqrt{2\pi}}e^{-(\log y)^2/2}\), \(y>0\) |
| Multivariate | \(f_{\mathbf Y}(\mathbf y)=f_{\mathbf X}(\mathbf x)\lvert\det\partial\mathbf x/\partial\mathbf y\rvert\) |
| Either direction | \(\lvert\det\partial\mathbf x/\partial\mathbf y\rvert=1/\lvert\det\partial\mathbf y/\partial\mathbf x\rvert\) |
| Discrete convolution | \(P(T=t)=\sum_xP(X=x)P(Y=t-x)\) |
| Continuous convolution | \(f_T(t)=\int f_X(x)f_Y(t-x)\,dx\), justified via the CDF |
| Preference order | Story, then MGF, then convolution sum or integral |
| Probabilistic method | \(P(A)>0\) for a random object, or some object scores at least \(E(X)\) |
| Committees | Expected overlap \(20/7\), so some pair shares at least 3 |
The factor \(\frac{N-n}{N-1}\) by which the Hypergeometric variance falls below the corresponding Binomial variance.
For a smooth invertible transformation, the PDF of the new variable is the old PDF times the absolute derivative, or absolute Jacobian determinant, of the inverse map.
The matrix of all partial derivatives of one set of variables with respect to another. The absolute value of its determinant rescales densities in a multivariate transformation.
The distribution of \(e^Z\) with \(Z\) Normal, so that its logarithm is Normal.
The distribution of a sum of independent random variables, given by \(\sum_xP(X=x)P(Y=t-x)\) or \(\int f_X(x)f_Y(t-x)\,dx\).
Proving that an object with a property exists by showing that a randomly chosen object has the property with positive probability, or that the average score is high enough.
The Beta distribution generalizes the Uniform: it lives on \((0,1)\), but its density \(x^{a-1}(1-x)^{b-1}\) can be flat, sloped, U-shaped, or hump-shaped depending on the parameters \(a,b>0\). Its main use is as a distribution for an unknown probability. It is the conjugate prior for the Binomial: a \(\operatorname{Beta}(a,b)\) prior and \(X\) successes in \(n\) trials give a \(\operatorname{Beta}(a+X,\,b+n-X)\) posterior, found without computing any normalizing constant. Bayes’ billiards evaluates \(\int_0^1\binom nk x^k(1-x)^{n-k}\,dx=\frac1{n+1}\) without calculus, which gives the Beta normalizing constant for integer parameters. A guest segment shows expectation at work in finance.
| Symbol | Meaning |
|---|---|
| \(X\sim\operatorname{Beta}(a,b)\) | Beta distribution with parameters \(a,b>0\) |
| \(\beta(a,b)\) | Beta function, \(\int_0^1x^{a-1}(1-x)^{b-1}\,dx\) |
| \(p\sim\operatorname{Beta}(a,b)\) | Prior distribution for an unknown probability |
| \(X\mid p\sim\operatorname{Bin}(n,p)\) | Data: successes in \(n\) trials, given \(p\) |
| \(f(p\mid X=k)\) | Posterior PDF of \(p\) |
| \(\propto\) | Proportional to, as a function of the variable of interest |
| \(S_T\), \(K\) | Asset price at a future time \(T\) and an option’s strike price |
After studying these notes, you should be able to:
So far the Uniform is the only named continuous distribution that is bounded, and its density is flat. The Normal lives on the whole line and the Exponential on the positive half-line. The Beta stays on \((0,1)\) but allows other shapes.
A random variable \(X\) has the Beta distribution with parameters \(a>0\) and \(b>0\), written \(X\sim\operatorname{Beta}(a,b)\), if its PDF is
\[\boxed{f(x)=c\,x^{a-1}(1-x)^{b-1},\qquad0<x<1,}\]and 0 otherwise. The normalizing constant is \(c=1/\beta(a,b)\), where \(\beta(a,b)=\int_0^1x^{a-1}(1-x)^{b-1}\,dx\) is the beta function.
It is a whole family, one distribution for each choice of \(a\) and \(b\), and the parameters need not be integers. Much can be done without knowing \(c\); its general value is found in Lecture 25.
A flexible family. Varying the parameters changes the shape:
| Parameters | Density | Shape |
|---|---|---|
| \(a=b=1\) | \(1\) | Flat: \(\operatorname{Beta}(1,1)=\operatorname{Unif}(0,1)\) |
| \(a=2,\ b=1\) | \(2x\) | Increases linearly; the triangle has area 1 |
| \(a=b=\tfrac12\) | \(\propto x^{-1/2}(1-x)^{-1/2}\) | U-shaped, blowing up at both ends |
| \(a=b=2\) | \(6x(1-x)\) | An upside-down U, symmetric about \(1/2\) |
In general, \(a\) pushes mass toward 1 and \(b\) pushes it toward 0. When \(a=b\) the density is symmetric about \(1/2\).
Why the Beta matters.
Laplace’s rule of succession (Lecture 17) treated an unknown probability \(p\) as Uniform. Replace the Uniform prior with any Beta prior.
The model.
\[p\sim\operatorname{Beta}(a,b),\qquad X\mid p\sim\operatorname{Bin}(n,p).\]Here \(p\) is treated as a random variable to reflect uncertainty about it, the Bayesian approach. The prior \(\operatorname{Beta}(a,b)\) describes beliefs about \(p\) before the data. \(X\) is the data. The goal is the posterior, the distribution of \(p\) given \(X\).
Bayes’ rule with a density.
\[f(p\mid X=k)=\frac{P(X=k\mid p)\,f(p)}{P(X=k)}.\]The denominator is a probability that does not depend on \(p\): it averages over all values of \(p\). As a function of \(p\), it is a constant. Constants such as \(\binom nk\) and \(c\) can be dropped too:
\[f(p\mid X=k)\propto p^k(1-p)^{n-k}\cdot p^{a-1}(1-p)^{b-1}=p^{a+k-1}(1-p)^{b+n-k-1}.\]This is the shape of a Beta density. The missing constant is whatever makes it integrate to 1, so the posterior must be a Beta:
Conjugacy. A family of priors is conjugate for a model if every prior in the family produces a posterior in the same family. Starting from a Beta, the data only update the parameters. This is computationally convenient: no new distribution has to be studied. Whether a conjugate prior is the right description of one’s uncertainty is a separate, debated question.
Interpretation. Read \(a\) and \(b\) as earlier, imagined data: \(a\) prior successes and \(b\) prior failures. The new data add \(X\) successes and \(n-X\) failures. Laplace’s rule is the case \(a=b=1\).
Suppose a prior \(p\sim\operatorname{Beta}(2,2)\), symmetric about \(1/2\) and mildly favoring middle values. In 10 independent trials there are 7 successes.
The posterior is \(\operatorname{Beta}(2+7,\ 2+3)=\operatorname{Beta}(9,5)\), with density proportional to \(p^8(1-p)^4\). It is concentrated near \(0.7\) and much narrower than the prior: the data have both moved and sharpened the belief.
If the same 10 trials are split into two batches, say 4 successes in 5 trials, then 3 in 5, the sequential updates give \(\operatorname{Beta}(6,3)\) and then \(\operatorname{Beta}(9,5)\), the same answer. This is the coherence of Bayes’ rule from Lecture 5.
Looking ahead. Moments of the Beta are easy once the normalizing constant is known: \(x\cdot x^{a-1}=x^{(a+1)-1}\), so \(E(X)\) is an integral of the same Beta form with \(a\) raised by 1, and \(E(X^2)\) raises \(a\) by 2. Lecture 25 finds the constant and the mean.
Incorrect: To find the posterior, the denominator \(P(X=k)\) must be computed first.
Correction: \(P(X=k)\) does not depend on \(p\). Work up to proportionality in \(p\), recognize the Beta shape, and the normalizing constant takes care of itself.
For integer parameters, Bayes found the needed integral without calculus. For integers \(0\le k\le n\), the goal is
\[\int_0^1\binom nk x^k(1-x)^{n-k}\,dx.\]Expanding by the binomial theorem and integrating term by term would be tedious. Instead, compare two ways of producing the same random picture.
Story 1. Take \(n+1\) white billiard balls and paint one pink. Throw all of them independently and uniformly onto \((0,1)\).
Story 2. Throw all \(n+1\) white balls independently and uniformly onto \((0,1)\), then choose one at random and paint it pink.
The two stories produce the same distribution of pictures: it does not matter whether the pink ball was chosen before or after the throw. Let \(X\) be the number of white balls to the left of the pink ball.
From Story 1. Throw the pink ball first and condition on its position \(p\). Each of the \(n\) white balls independently lands left of the pink ball with probability \(p\), so \(X\mid p\sim\operatorname{Bin}(n,p)\). The pink ball’s position is Uniform, with density 1. By the continuous law of total probability,
\[P(X=k)=\int_0^1\binom nk p^k(1-p)^{n-k}\,dp.\]From Story 2. The pink ball is a uniformly random one of the \(n+1\) thrown balls, so its rank among them is equally likely to be any of \(1,\ldots,n+1\). The number of white balls to its left is therefore equally likely to be any of \(0,1,\ldots,n\):
\[P(X=k)=\frac1{n+1}.\]Equivalently, \(\beta(k+1,\,n-k+1)=\dfrac{k!\,(n-k)!}{(n+1)!}\). This is the Beta normalizing constant whenever both parameters are positive integers.
Find \(\int_0^1x^2(1-x)^3\,dx\). Here \(k=2\) and \(n=5\), so
\[\int_0^1x^2(1-x)^3\,dx=\frac1{\binom52\cdot6}=\frac1{60}.\]So the \(\operatorname{Beta}(3,4)\) density is \(60x^2(1-x)^3\).
This part summarizes a guest presentation on Statistics 123, Applied Quantitative Finance, a course whose only prerequisite is Statistics 110. It shows how expectation and LOTUS appear in quantitative finance.
Financial derivatives. A financial derivative is a contract whose payoff at a future date is a function of another random quantity, usually an asset price \(S_T\) at time \(T\). A contract paying $1 if Boston gets more than 80 inches of snow this winter is a weather derivative. In finance the payoff is a random variable \(g(S_T)\), and a key result says that, with the distribution of \(S_T\) chosen appropriately, the fair price is an expectation:
\[\text{price}=E\big(g(S_T)\big)=\int g(s)f(s)\,ds,\]which is LOTUS.
Suppose one euro is worth $1 today, and in a year it will be worth either $1.25 or $0.80, each with probability \(1/2\). The expected value of a euro in a year is
\[\frac12(1.25)+\frac12(0.80)=1.025\text{ dollars},\]so the euro is expected to appreciate. Now describe the same model from the dollar’s side. A dollar will be worth either \(1/1.25=0.80\) euros or \(1/0.80=1.25\) euros, each with probability \(1/2\), so its expected value is also \(1.025\) euros. The dollar is expected to appreciate against the euro too.
Both calculations are correct. They concern different random variables, \(S\) and \(1/S\), and \(E(1/S)\ne1/E(S)\) because \(1/s\) is not linear. Expectation does not commute with nonlinear functions, as the St. Petersburg paradox showed with \(E(2^X)\ne2^{E(X)}\). Deciding which expectation should be used to price currency contracts is a deeper question, taken up in Statistics 123.
In October 2008, as part of the government’s response to the financial crisis, the US government paid Goldman Sachs about $450 million. In return it received warrants: the right, but not the obligation, to buy 10 million Goldman Sachs shares at $125 each within ten years. The shares were then trading at about $95.
A warrant of this kind works like a call option with strike price \(K=125\). At maturity, it is used only if the share price is above $125, so its value is
\[g(S_T)=\max(S_T-K,\,0).\]If the price is $150, it is worth $25; at $100, it is worth $0. The implied price was \(450\text{ million}/10\text{ million}=\$45\) per option. With a density \(f\) for \(S_T\), LOTUS gives
\[E\big(\max(S_T-K,0)\big)=\int_K^\infty(s-K)f(s)\,ds.\]Under a particular Log-Normal model for \(S_T\), this integral is the Black–Scholes formula.
Try these before reading the answers.
\(\int_0^1x^2\,dx=\frac13\), so \(f(x)=3x^2\). Then \(P(X>1/2)=1-(1/2)^3=\frac78\).
\(f(x)=6x(1-x)\), with CDF \(3x^2-2x^3\). So \(P(X<1/3)=\frac13-\frac2{27}=\frac7{27}\).
The prior is \(\operatorname{Beta}(1,1)\), so the posterior is \(\operatorname{Beta}(1+3,\,1+1)=\operatorname{Beta}(4,2)\).
After a success: \(\operatorname{Beta}(2,1)\). After another success: \(\operatorname{Beta}(3,1)\). After a failure: \(\operatorname{Beta}(3,2)\). Any order gives 2 successes and 1 failure, hence \(\operatorname{Beta}(3,2)\).
With \(k=3\) and \(n=5\), \(\binom53\int_0^1x^3(1-x)^2\,dx=\frac16\), so the integral is \(\frac1{60}\), matching \(\frac{3!\,2!}{6!}\).
Let \(Y=1-X\), a decreasing map with \(\lvert dx/dy\rvert=1\). Then \(f_Y(y)=c(1-y)^{a-1}y^{b-1}\), the \(\operatorname{Beta}(b,a)\) shape.
Maximize \((a-1)\log x+(b-1)\log(1-x)\). Setting the derivative \(\frac{a-1}x-\frac{b-1}{1-x}\) to 0 gives \(x=\frac{a-1}{a+b-2}\). For the posterior \(\operatorname{Beta}(9,5)\) in the worked example, the mode is \(8/12=2/3\).
\(E(S)=1.025\), so \(1/E(S)\approx0.976\). But \(E(1/S)=\frac12(0.80)+\frac12(1.25)=1.025\). They differ because \(1/s\) is not linear.
\(E(\max(S-125,0))=\int_{125}^{150}(s-125)\frac1{50}\,ds=\frac{25^2/2}{50}=6.25\) dollars.
| Concept | Essential fact |
|---|---|
| Beta PDF | \(c\,x^{a-1}(1-x)^{b-1}\) on \((0,1)\), with \(a,b>0\) |
| Special cases | \(\operatorname{Beta}(1,1)=\operatorname{Unif}(0,1)\); \(\operatorname{Beta}(2,1)\) has density \(2x\) |
| Shapes | U-shaped if \(a,b<1\); hump-shaped if \(a,b>1\); symmetric if \(a=b\) |
| Main use | Prior for an unknown probability |
| Model | \(p\sim\operatorname{Beta}(a,b)\), \(X\mid p\sim\operatorname{Bin}(n,p)\) |
| Posterior | \(p\mid X\sim\operatorname{Beta}(a+X,\,b+n-X)\) |
| Method | Work up to proportionality in \(p\); the denominator is a constant |
| Conjugacy | Beta prior gives Beta posterior; parameters act as prior successes and failures |
| Bayes’ billiards | \(\int_0^1\binom nk x^k(1-x)^{n-k}dx=\frac1{n+1}\) |
| Integer constant | \(\beta(k+1,n-k+1)=\frac{k!(n-k)!}{(n+1)!}\) |
| Derivative price | Expectation of the payoff \(g(S_T)\), by LOTUS |
| Call option | Payoff \(\max(S_T-K,0)\) |
| Exchange-rate puzzle | \(E(1/S)\ne1/E(S)\) |
A continuous distribution on \((0,1)\) with density proportional to \(x^{a-1}(1-x)^{b-1}\), generalizing the Uniform.
\(\beta(a,b)=\int_0^1x^{a-1}(1-x)^{b-1}\,dx\), the reciprocal of the Beta normalizing constant.
The distribution of an unknown parameter before observing data, and its updated distribution after observing the data via Bayes’ rule.
A family of prior distributions for which the posterior, after observing data from a given model, stays in the same family. The Beta is conjugate for the Binomial.
The argument that painting one of \(n+1\) uniformly thrown balls before or after the throw gives the same picture, which evaluates \(\int_0^1\binom nk x^k(1-x)^{n-k}dx=\frac1{n+1}\).
A contract whose payoff is a function of another random quantity, such as an asset price. A call option pays \(\max(S_T-K,0)\).
Factorials grow roughly like \((n/e)^n\), made precise by Stirling’s formula. The gamma function \(\Gamma(a)=\int_0^\infty x^ae^{-x}\frac{dx}x\) extends the factorial to all positive reals, with \(\Gamma(n)=(n-1)!\), \(\Gamma(a+1)=a\Gamma(a)\), and \(\Gamma(\tfrac12)=\sqrt\pi\). Dividing its integrand by \(\Gamma(a)\) gives the \(\operatorname{Gamma}(a,1)\) PDF, and rescaling gives \(\operatorname{Gamma}(a,\lambda)\). In a Poisson process of rate \(\lambda\), the gaps between arrivals are i.i.d. \(\operatorname{Expo}(\lambda)\), and the time of the \(n\)th arrival, a sum of \(n\) such gaps, is \(\operatorname{Gamma}(n,\lambda)\), proved by matching MGFs. The Gamma is to the Exponential what the Negative Binomial is to the Geometric. Its moments come straight from the gamma function: a \(\operatorname{Gamma}(a,\lambda)\) variable has mean \(a/\lambda\) and variance \(a/\lambda^2\).
| Symbol | Meaning |
|---|---|
| \(\Gamma(a)\) | Gamma function, \(\int_0^\infty x^ae^{-x}\frac{dx}x\) for \(a>0\) |
| \(X\sim\operatorname{Gamma}(a,1)\) | PDF \(\frac1{\Gamma(a)}x^ae^{-x}\frac1x\), \(x>0\) |
| \(Y\sim\operatorname{Gamma}(a,\lambda)\) | \(Y=X/\lambda\) with \(X\sim\operatorname{Gamma}(a,1)\) |
| \(N_t\) | Number of arrivals in \([0,t]\) in a Poisson process, \(N_t\sim\operatorname{Pois}(\lambda t)\) |
| \(T_n\) | Time of the \(n\)th arrival |
| \(X_j\) | Gap between arrivals \(j-1\) and \(j\), with \(T_0=0\) |
| \(M(t)\) | Moment generating function |
After studying these notes, you should be able to:
A sequence such as \(0,1,2,\ldots\) can be continued in many ways. The next term could be 3, or it could be \(3!!!\) if the pattern is \(0\), \(1!\), \(2!!\) with repeated factorials. Without a principle for choosing, such puzzles have no single answer. The same problem arises in extending the factorial beyond the integers: many smooth curves pass through the points \((n,n!)\). The gamma function (Part 2) is the standard choice.
First, the size of \(n!\). The values \(1,1,2,6,24,120,720,\ldots\) start small and then grow extremely fast. Stirling’s formula describes the growth:
in the sense that the ratio of the two sides tends to 1 as \(n\to\infty\).
The dominant part is \(n^n\), discounted by \(e^n\); the square root is a smaller correction. The approximation is accurate even for modest \(n\): for \(n=10\), it gives about \(3{,}598{,}696\), against \(10!=3{,}628{,}800\), a ratio of about \(0.992\). Stirling’s formula can be proved with probability, using the Poisson distribution, but that proof is beyond this lecture.
For real \(a>0\),
\[\boxed{\Gamma(a)=\int_0^\infty x^ae^{-x}\,\frac{dx}x.}\]This is the same as \(\int_0^\infty x^{a-1}e^{-x}\,dx\). Keeping \(\frac{dx}x\) separate is convenient: under a rescaling \(u=cx\), \(\frac{du}u=\frac{dx}x\), so it is unchanged.
Why \(a>0\). Near \(x=0\), \(e^{-x}\approx1\), so the integrand behaves like \(x^{a-1}\), which is integrable near 0 exactly when \(a>0\). With \(a=0\), the integrand near 0 is about \(1/x\), whose integral \(\log x\) diverges. As \(x\to\infty\), exponential decay beats any power, so that end is always fine.
Key properties.
Incorrect: \(\Gamma(n)=n!\).
Correction: \(\Gamma(n)=(n-1)!\). For example, \(\Gamma(1)=0!=1\) and \(\Gamma(5)=4!=24\).
Dividing the definition of \(\Gamma(a)\) by \(\Gamma(a)\) gives an integrand that is nonnegative and integrates to 1, so it is a PDF:
\[1=\int_0^\infty\frac1{\Gamma(a)}x^ae^{-x}\frac{dx}x.\]\(X\sim\operatorname{Gamma}(a,1)\) if its PDF is
\[\boxed{f_X(x)=\frac1{\Gamma(a)}x^ae^{-x}\frac1x,\qquad x>0.}\]For \(\lambda>0\), \(Y\sim\operatorname{Gamma}(a,\lambda)\) if \(Y=X/\lambda\) with \(X\sim\operatorname{Gamma}(a,1)\).
The \(\operatorname{Gamma}(a,\lambda)\) PDF. This is a rescaling, as with the Exponential. With \(y=x/\lambda\), \(x=\lambda y\) and \(\frac{dx}{dy}=\lambda\):
\[f_Y(y)=f_X(\lambda y)\cdot\lambda=\frac1{\Gamma(a)}(\lambda y)^ae^{-\lambda y}\frac1{\lambda y}\cdot\lambda =\boxed{\frac1{\Gamma(a)}(\lambda y)^ae^{-\lambda y}\frac1y,\qquad y>0.}\]With \(a=1\), this is \(\lambda e^{-\lambda y}\), so \(\operatorname{Gamma}(1,\lambda)=\operatorname{Expo}(\lambda)\). It is easy to forget the factor \(\frac1y\); writing \(y^{a-1}\) instead of \(y^a\cdot\frac1y\) is equivalent.
So far the Gamma is only a convenient normalization of a famous integral. Its importance comes from its stories.
Imagine emails arriving in an inbox. A Poisson process of rate \(\lambda\) satisfies two conditions:
The first arrival. Waiting longer than \(t\) for the first email means no emails arrived by time \(t\):
\[P(T_1>t)=P(N_t=0)=e^{-\lambda t}.\]This is the survival function of \(\operatorname{Expo}(\lambda)\), so \(T_1\sim\operatorname{Expo}(\lambda)\).
The later gaps. By the memoryless property and independence over disjoint intervals, after each arrival the process starts afresh. The gap from the first arrival to the second is again \(\operatorname{Expo}(\lambda)\), independent of the first, and so on.
The interarrival times \(X_1,X_2,\ldots\) are i.i.d. \(\operatorname{Expo}(\lambda)\), and the time of the \(n\)th arrival is
\[\boxed{T_n=X_1+\cdots+X_n\sim\operatorname{Gamma}(n,\lambda).}\]The i.i.d. Exponential gaps are an equivalent way to define a Poisson process. The Gamma claim is proved in Part 5.
An analogy.
| Discrete time | Continuous time |
|---|---|
| Geometric: wait for the first success | Exponential: wait for the first arrival |
| Negative Binomial: sum of i.i.d. Geometrics | Gamma: sum of i.i.d. Exponentials |
The Exponential is the continuous analog of the Geometric, and the Gamma is the continuous analog of the Negative Binomial: how long to wait for \(n\) “successes”, here \(n\) arrivals.
Sums of i.i.d. variables could be handled with repeated convolution integrals, but MGFs are much easier. Rescaling by \(\lambda\) can be done at the end, so take \(\lambda=1\).
The sum of Exponentials. In Lecture 18, the \(\operatorname{Expo}(1)\) MGF was found to be \(\frac1{1-t}\) for \(t<1\). For independent variables, MGFs multiply, so
\[M_{T_n}(t)=\left(\frac1{1-t}\right)^n,\qquad t<1.\]The Gamma MGF. Let \(Y\sim\operatorname{Gamma}(a,1)\). By LOTUS,
\[M_Y(t)=E(e^{tY})=\frac1{\Gamma(a)}\int_0^\infty e^{ty}y^ae^{-y}\frac{dy}y =\frac1{\Gamma(a)}\int_0^\infty y^ae^{-(1-t)y}\frac{dy}y.\]For \(t<1\), substitute \(x=(1-t)y\). The limits stay \(0\) to \(\infty\), \(y^a=(1-t)^{-a}x^a\), and \(\frac{dy}y=\frac{dx}x\):
\[M_Y(t)=(1-t)^{-a}\cdot\frac1{\Gamma(a)}\int_0^\infty x^ae^{-x}\frac{dx}x=\left(\frac1{1-t}\right)^a.\]With \(a=n\), this matches the MGF of \(T_n\). Since an MGF determines the distribution, \(T_n\sim\operatorname{Gamma}(n,1)\). Dividing by \(\lambda\) gives the general result. \(\blacksquare\)
The Gamma MGF calculation never used that \(a\) is an integer: \(\left(\frac1{1-t}\right)^a\) is the \(\operatorname{Gamma}(a,1)\) MGF for every \(a>0\). Only the sum-of-Exponentials story needs an integer.
Core idea: Recognizing an integral as a gamma function, after a change of variable, replaces integration by pattern matching.
Let \(X\sim\operatorname{Gamma}(a,1)\) and let \(c\) be any real number with \(a+c>0\). By LOTUS,
\[E(X^c)=\frac1{\Gamma(a)}\int_0^\infty x^cx^ae^{-x}\frac{dx}x=\frac1{\Gamma(a)}\int_0^\infty x^{a+c}e^{-x}\frac{dx}x.\]The last integral is \(\Gamma(a+c)\), so
If \(a+c\le0\), the moment does not exist.
Mean and variance. With \(c=1\) and the recursion,
\[E(X)=\frac{\Gamma(a+1)}{\Gamma(a)}=a,\qquad E(X^2)=\frac{\Gamma(a+2)}{\Gamma(a)}=(a+1)a,\]so \(\operatorname{Var}(X)=a^2+a-a^2=a\). For integer \(a\), the mean also follows from linearity: a sum of \(a\) i.i.d. \(\operatorname{Expo}(1)\) variables has mean \(a\).
Rescaling by \(1/\lambda\) divides the mean by \(\lambda\) and the variance by \(\lambda^2\):
The equal mean and variance of \(\operatorname{Gamma}(a,1)\) come from setting \(\lambda=1\); unlike for the Poisson, they differ in general.
Emails arrive as a Poisson process at \(\lambda=2\) per hour. The time of the third email is \(T_3\sim\operatorname{Gamma}(3,2)\), so
\[E(T_3)=\frac32\text{ hours},\qquad\operatorname{Var}(T_3)=\frac3{4}.\]Supplementary connection. Waiting more than one hour for the third email means at most two emails arrived in the first hour, and \(N_1\sim\operatorname{Pois}(2)\):
\[P(T_3>1)=P(N_1\le2)=e^{-2}\left(1+2+\frac{2^2}{2!}\right)=5e^{-2}\approx0.677.\]Try these before reading the answers.
\(\Gamma(5)=4!=24\). \(\Gamma(7/2)=\frac52\cdot\frac32\cdot\frac12\sqrt\pi=\frac{15\sqrt\pi}8\approx3.323\).
With \(u=x^a\) and \(dv=e^{-x}dx\), \(\Gamma(a+1)=\int_0^\infty x^ae^{-x}dx=\big[-x^ae^{-x}\big]_0^\infty+a\int_0^\infty x^{a-1}e^{-x}dx=0+a\Gamma(a)\).
Mean \(a/\lambda=4/2=2\), variance \(a/\lambda^2=4/4=1\).
With \(c=-1\), \(E(X^{-1})=\frac{\Gamma(a-1)}{\Gamma(a)}=\frac{\Gamma(a-1)}{(a-1)\Gamma(a-1)}=\frac1{a-1}\). It requires \(a-1>0\).
The \(\operatorname{Gamma}(a,\lambda)\) MGF is \(\left(\frac{\lambda}{\lambda-t}\right)^a\) for \(t<\lambda\), from rescaling. The product is \(\left(\frac\lambda{\lambda-t}\right)^{a+b}\), so \(X+Y\sim\operatorname{Gamma}(a+b,\lambda)\). For integers, this is \(a\) gaps followed by \(b\) more.
\(T_2\sim\operatorname{Gamma}(2,3)\). The second customer arrives within a minute exactly when \(N_1\ge2\), with \(N_1\sim\operatorname{Pois}(3)\): \(P(T_2\le1)=1-e^{-3}(1+3)=1-4e^{-3}\approx0.801\).
\(\sqrt{10\pi}(5/e)^5\approx118.0\), against \(5!=120\), a relative error under 2%.
For \(a\le0\) and \(0<x<1\), the integrand is at least \(e^{-1}x^{a-1}\ge e^{-1}x^{-1}\), and \(\int_0^1x^{-1}dx=\infty\).
\(\operatorname{Expo}(1)=\operatorname{Gamma}(1,1)\), so \(E(X^n)=\Gamma(n+1)/\Gamma(1)=n!\), the same answer as reading off the geometric series in Lecture 18.
| Concept | Essential fact |
|---|---|
| Stirling | \(n!\approx\sqrt{2\pi n}(n/e)^n\), with ratio tending to 1 |
| Gamma function | \(\Gamma(a)=\int_0^\infty x^ae^{-x}dx/x\), \(a>0\) |
| Recursion | \(\Gamma(a+1)=a\Gamma(a)\) |
| Factorials | \(\Gamma(n)=(n-1)!\) |
| Half | \(\Gamma(1/2)=\sqrt\pi\) |
| \(\operatorname{Gamma}(a,1)\) PDF | \(\frac1{\Gamma(a)}x^ae^{-x}\frac1x\), \(x>0\) |
| \(\operatorname{Gamma}(a,\lambda)\) PDF | \(\frac1{\Gamma(a)}(\lambda y)^ae^{-\lambda y}\frac1y\), \(y>0\) |
| Exponential | \(\operatorname{Gamma}(1,\lambda)=\operatorname{Expo}(\lambda)\) |
| Poisson process | \(N_t\sim\operatorname{Pois}(\lambda t)\); disjoint intervals independent |
| Gaps | i.i.d. \(\operatorname{Expo}(\lambda)\) |
| Arrival times | \(T_n\sim\operatorname{Gamma}(n,\lambda)\) |
| MGF | \(\operatorname{Gamma}(a,1)\): \((1-t)^{-a}\) for \(t<1\) |
| Moments | \(E(X^c)=\Gamma(a+c)/\Gamma(a)\) for \(\operatorname{Gamma}(a,1)\) |
| Mean and variance | \(a/\lambda\) and \(a/\lambda^2\) |
The approximation \(n!\approx\sqrt{2\pi n}(n/e)^n\), whose ratio to \(n!\) tends to 1.
\(\Gamma(a)=\int_0^\infty x^ae^{-x}\frac{dx}x\) for \(a>0\), extending the factorial through \(\Gamma(n)=(n-1)!\).
The continuous distribution on \((0,\infty)\) with PDF \(\frac1{\Gamma(a)}(\lambda y)^ae^{-\lambda y}\frac1y\). For integer \(a\), it is the distribution of a sum of \(a\) i.i.d. \(\operatorname{Expo}(\lambda)\) variables.
A model of arrivals in time in which counts in an interval of length \(t\) are \(\operatorname{Pois}(\lambda t)\) and counts in disjoint intervals are independent.
The time between consecutive arrivals. In a Poisson process of rate \(\lambda\), these are i.i.d. \(\operatorname{Expo}(\lambda)\).
If \(X\sim\operatorname{Gamma}(a,\lambda)\) and \(Y\sim\operatorname{Gamma}(b,\lambda)\) are independent waiting times at a bank and a post office, then the total \(X+Y\) and the fraction \(X/(X+Y)\) spent at the bank are independent, with distributions \(\operatorname{Gamma}(a+b,\lambda)\) and \(\operatorname{Beta}(a,b)\). A Jacobian calculation proves this and, as a by-product, gives the Beta normalizing constant \(\frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)}\). The same independence gives the Beta mean \(\frac a{a+b}\) in one line. Order statistics, the sample sorted from smallest to largest, are dependent even for i.i.d. data; a Binomial argument gives the CDF of the \(j\)th one, and a picture gives its PDF. Uniform order statistics are Beta. Conditional expectation extends the law of total probability to averages, and the two-envelope paradox shows why it needs care.
| Symbol | Meaning |
|---|---|
| \(X\sim\operatorname{Gamma}(a,\lambda)\), \(Y\sim\operatorname{Gamma}(b,\lambda)\) | Independent waits at the bank and the post office |
| \(T=X+Y\) | Total waiting time |
| \(W=\dfrac X{X+Y}\) | Fraction of the total spent at the bank |
| \(\beta(a,b)=\dfrac{\Gamma(a)\Gamma(b)}{\Gamma(a+b)}\) | Beta function, in terms of gamma functions |
| \(X_{(1)}\le X_{(2)}\le\cdots\le X_{(n)}\) | Order statistics: minimum to maximum |
| \(F\), \(f\) | Common CDF and PDF of i.i.d. continuous \(X_1,\ldots,X_n\) |
| \(E(X\mid A)\) | Expectation of \(X\) computed with probabilities conditional on event \(A\) |
After studying these notes, you should be able to:
You wait \(X\) minutes at the bank and then \(Y\) minutes at the post office, with
\[X\sim\operatorname{Gamma}(a,\lambda),\qquad Y\sim\operatorname{Gamma}(b,\lambda),\qquad X,Y\text{ independent}.\]For integer \(a\), a natural interpretation is that \(a\) people are ahead of you in line, each served in an independent \(\operatorname{Expo}(\lambda)\) time. Consider
\[T=X+Y\ \text{(total wait)},\qquad W=\frac X{X+Y}\ \text{(fraction at the bank)}.\]The marginal distribution of \(T\) is already known: \(a\) Exponential gaps plus \(b\) more is \(a+b\) gaps, so \(T\sim\operatorname{Gamma}(a+b,\lambda)\), and MGFs give the same for non-integer parameters. Are \(T\) and \(W\) independent? The formula \(W=X/(X+Y)\) contains \(T\), which suggests dependence, but it is not obvious that knowing the total says anything about the fraction. A calculation settles it.
Set \(\lambda=1\). Rescaling all times by \(1/\lambda\) leaves \(W\) unchanged and turns \(\operatorname{Gamma}(a+b,1)\) into \(\operatorname{Gamma}(a+b,\lambda)\), so nothing is lost.
Invert the transformation. With \(t=x+y\) and \(w=x/(x+y)\), we have \(w=x/t\), so \(x=tw\). Also \(1-w=y/t\), so \(y=t(1-w)\).
The Jacobian.
\[\frac{\partial(x,y)}{\partial(t,w)}= \begin{pmatrix}w&t\\1-w&-t\end{pmatrix},\qquad \det=-tw-t(1-w)=-t,\qquad\lvert\det\rvert=t.\]The joint PDF. Independence gives \(f_{X,Y}(x,y)=\frac1{\Gamma(a)\Gamma(b)}x^ae^{-x}y^be^{-y}\frac1{xy}\). Substitute \(x=tw\), \(y=t(1-w)\), \(x+y=t\), and multiply by \(t\):
\[f_{T,W}(t,w)=\frac1{\Gamma(a)\Gamma(b)}\,w^{a-1}(1-w)^{b-1}\cdot t^{a+b}e^{-t}\frac1t, \qquad t>0,\ 0<w<1.\]Multiplying and dividing by \(\Gamma(a+b)\) to make the \(t\) part a familiar density:
The joint PDF factors into a function of \(w\) times a function of \(t\) on a product region, so \(T\) and \(W\) are independent. The second factor is the \(\operatorname{Gamma}(a+b,1)\) PDF. Integrating out \(t\) removes it, since it integrates to 1, leaving the marginal PDF of \(W\).
The Beta normalizing constant. The marginal of \(W\) must integrate to 1, so its constant is the Beta normalizing constant:
For integers this agrees with Bayes’ billiards: \(\beta(k+1,n-k+1)=\frac{k!\,(n-k)!}{(n+1)!}\). The constant came out of a different question, without directly attacking the Beta integral. This independence is special: it is a theorem, beyond these notes, that replacing the Gamma by other distributions breaks it.
Let \(W\sim\operatorname{Beta}(a,b)\).
Method 1: LOTUS. Multiplying the density by \(w\) raises \(a\) by 1, leaving a Beta integral:
\[E(W)=\frac{\beta(a+1,b)}{\beta(a,b)}=\frac{\Gamma(a+1)\Gamma(b)}{\Gamma(a+b+1)}\cdot\frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)}=\frac a{a+b}.\]Higher moments work the same way: \(E(W^c)\) raises \(a\) by \(c\).
Method 2: the bank–post office story. The mean depends only on the distribution, so any construction of a \(\operatorname{Beta}(a,b)\) variable may be used. Take \(W=X/(X+Y)\) as in Part 1. Since \(W\) and \(T=X+Y\) are independent, they are uncorrelated, so \(E(WT)=E(W)E(T)\). But \(WT=X\), so
\[E(X)=E(W)E(X+Y),\qquad\text{so}\qquad E(W)=\frac{E(X)}{E(X+Y)}=\frac a{a+b}.\]The variance follows from LOTUS in the same way: \(E(W^2)=\frac{a(a+1)}{(a+b)(a+b+1)}\) and \(\operatorname{Var}(W)=\frac{ab}{(a+b)^2(a+b+1)}\).
In the Beta–Binomial model of Lecture 23, the posterior mean is \(\frac{a+X}{a+b+n}\). With \(a=b=1\) and \(X=n\), this is \(\frac{n+1}{n+2}\), Laplace’s rule of succession.
Incorrect: \(E\left(\dfrac X{X+Y}\right)=\dfrac{E(X)}{E(X+Y)}\) by linearity.
Correction: Linearity does not apply to ratios, and this equation is usually false. It holds for independent Gammas only because the fraction is independent of the total. For example, if \(X\sim\operatorname{Unif}(0,1)\) and \(Y=1\), then \(E\left(\frac X{X+1}\right)=1-\log2\approx0.307\), but \(\frac{E(X)}{E(X+1)}=\frac13\).
For random variables \(X_1,\ldots,X_n\), the order statistics are the same values sorted in increasing order:
\[X_{(1)}\le X_{(2)}\le\cdots\le X_{(n)}.\]So \(X_{(1)}=\min(X_1,\ldots,X_n)\), \(X_{(n)}=\max(X_1,\ldots,X_n)\), and \(X_{(j)}\) is the \(j\)th smallest. Each \(X_{(j)}\) is a random variable: for each outcome of the experiment, compute the values and sort them.
Many statistics are built from order statistics:
Why they are hard. Even when \(X_1,\ldots,X_n\) are i.i.d., the order statistics are dependent. The maximum is at least the minimum, so learning that the minimum is large forces the maximum to be large. In the same way, \(X_{(7)}\ge X_{(3)}\) always. Order statistics are positively correlated.
Ties. For discrete variables, two observations can be equal, which complicates the ordering. For continuous variables, ties have probability 0. From here on, \(X_1,\ldots,X_n\) are i.i.d. continuous with CDF \(F\) and PDF \(f\).
Incorrect: The minimum and maximum of two i.i.d. variables are just \(X\) and \(Y\) under different names, so \(\operatorname{Cov}(\min,\max)=\operatorname{Cov}(X,Y)=0\).
Correction: Which variable is the minimum changes from outcome to outcome. The minimum and maximum are new random variables, and they are positively correlated.
The CDF. Draw a number line and mark a point \(x\). The event \(X_{(j)}\le x\) means that the \(j\)th smallest value lies to the left of \(x\). Then the \(j-1\) smaller ones do too, and possibly more. In words:
\[X_{(j)}\le x\quad\Longleftrightarrow\quad\text{at least }j\text{ of the }X_i\text{ are }\le x.\]Call \(X_i\le x\) a success. The \(n\) observations are independent, each succeeding with probability \(F(x)\), so the number of successes is \(\operatorname{Bin}(n,F(x))\). Adding the disjoint cases of exactly \(k\) successes:
The PDF. Differentiating this sum works but is messy. Think directly instead. \(f_{X_{(j)}}(x)\,dx\) is approximately the probability that \(X_{(j)}\) falls in a tiny interval of length \(dx\) at \(x\). For that to happen:
Multiplying and dropping \(dx\) from both sides:
The formula combines the PDF, a power of the CDF, and a power of one minus the CDF. Joint PDFs of two order statistics, such as \(X_{(3)}\) and \(X_{(7)}\), can be found with the same picture using two tiny intervals.
Let \(U_1,\ldots,U_n\) be i.i.d. \(\operatorname{Unif}(0,1)\). Then \(f(x)=1\) and \(F(x)=x\) on \((0,1)\), so
\[f_{U_{(j)}}(x)=n\binom{n-1}{j-1}x^{j-1}(1-x)^{n-j},\qquad0<x<1.\]This has the Beta shape with parameters \(j\) and \(n-j+1\), so
\[\boxed{U_{(j)}\sim\operatorname{Beta}(j,\,n-j+1),\qquad E(U_{(j)})=\frac j{n+1}.}\]On average, the \(n\) points split \((0,1)\) into \(n+1\) pieces of equal expected length.
A check with \(n=2\). The maximum is \(\operatorname{Beta}(2,1)\) with mean \(2/3\), and the minimum is \(\operatorname{Beta}(1,2)\) with mean \(1/3\). Since \(\lvert U_1-U_2\rvert=\max-\min\), linearity gives
\[E\lvert U_1-U_2\rvert=\frac23-\frac13=\frac13,\]matching the two-dimensional LOTUS calculation in Lecture 19. The maximum and minimum are dependent, but linearity holds regardless.
The conditional expectation \(E(X\mid A)\) of \(X\) given an event \(A\) is computed exactly like \(E(X)\), but with conditional probabilities given \(A\). In the discrete case, \(E(X\mid A)=\sum_xxP(X=x\mid A)\).
Splitting each \(P(X=x)\) by the law of total probability gives the analogous rule for expectations:
Roll a fair die. Given an even result, the values \(2,4,6\) are equally likely, so \(E(X\mid\text{even})=4\). Given an odd result, \(E(X\mid\text{odd})=3\). Then
\[E(X)=4\cdot\frac12+3\cdot\frac12=3.5.\]The two-envelope paradox. Two identical-looking envelopes contain positive amounts of money, one exactly twice the other. You pick one at random and may switch. Suppose you open yours and find $100. The other holds $50 or $200, which seem equally likely, so switching seems to give an average of $125. The argument did not use the number 100: whatever amount \(x\) is in your envelope, the other appears to average \(\frac12\cdot\frac x2+\frac12\cdot2x=\frac54x\). Then you should switch without opening anything, and by the same reasoning switch back, forever. By symmetry, though, the two envelopes are equally good. Something in the switching argument is wrong; finding the flaw requires conditional expectation, developed in the next lecture.
Try these before reading the answers.
\(\frac{\Gamma(8)}{\Gamma(3)\Gamma(5)}=\frac{7!}{2!\,4!}=\frac{5040}{48}=105\). So the PDF is \(105x^2(1-x)^4\).
\(E=\frac3{3+5}=\frac38\). \(\operatorname{Var}=\frac{3\cdot5}{8^2\cdot9}=\frac{15}{576}=\frac5{192}\).
\(X\sim\operatorname{Gamma}(2,1)\) and \(Y\sim\operatorname{Gamma}(3,1)\), so \(W\sim\operatorname{Beta}(2,3)\) with PDF \(12w(1-w)^2\). Then
\[P(W<\tfrac12)=12\int_0^{1/2}(w-2w^2+w^3)\,dw=12\left(\frac18-\frac1{12}+\frac1{64}\right)=\frac{11}{16}.\]\(U_{(1)}\sim\operatorname{Beta}(1,n)\) with mean \(\frac1{n+1}\), and \(U_{(n)}\sim\operatorname{Beta}(n,1)\) with mean \(\frac n{n+1}\).
| The median is \(U_{(3)}\sim\operatorname{Beta}(3,3)\) with PDF \(30x^2(1-x)^2\). Integrating from 0 to \(1/4\) gives $$30\left(\frac{x^3}3-\frac{x^4}2+\frac{x^5}5\right)\Big | _0^{1/4}=\frac{53}{512}\approx0.104$$. |
With \(j=1\): \(f(x)=n\,e^{-x}\left(e^{-x}\right)^{n-1}=ne^{-nx}\), so the minimum is \(\operatorname{Expo}(n)\).
With \(j=n\), only the \(k=n\) term remains: \(F(x)^n\). Directly, the maximum is at most \(x\) exactly when every observation is at most \(x\), and independence multiplies the probabilities.
\(E(X\mid X\le2)=1.5\) with probability \(\frac13\), and \(E(X\mid X>2)=4.5\) with probability \(\frac23\). So \(E(X)=1.5\cdot\frac13+4.5\cdot\frac23=3.5\).
With \(X\sim\operatorname{Unif}(0,1)\) and \(Y=1\): \(E\left(\frac X{X+1}\right)=\int_0^1\left(1-\frac1{x+1}\right)dx=1-\log2\approx0.307\), while \(\frac{E(X)}{E(X+1)}=\frac{1/2}{3/2}=\frac13\).
| Concept | Essential fact |
|---|---|
| Bank–post office | \(X\sim\operatorname{Gamma}(a,\lambda)\), \(Y\sim\operatorname{Gamma}(b,\lambda)\) independent |
| Total and fraction | \(T\sim\operatorname{Gamma}(a+b,\lambda)\), \(W\sim\operatorname{Beta}(a,b)\), independent |
| Jacobian | \(x=tw\), \(y=t(1-w)\), \(\lvert\det\rvert=t\) |
| Beta constant | \(\frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)}\) |
| Beta mean | \(\frac a{a+b}\), by LOTUS or by independence of \(W\) and \(T\) |
| Beta variance | \(\frac{ab}{(a+b)^2(a+b+1)}\) |
| Ratio warning | \(E(X/(X+Y))=E(X)/E(X+Y)\) only with special structure |
| Order statistics | Sorted values; dependent, positively correlated |
| CDF | \(P(X_{(j)}\le x)=\sum_{k=j}^n\binom nkF(x)^k(1-F(x))^{n-k}\) |
| \(n\binom{n-1}{j-1}f(x)F(x)^{j-1}(1-F(x))^{n-j}\) | |
| Uniform case | \(U_{(j)}\sim\operatorname{Beta}(j,n-j+1)\), mean \(\frac j{n+1}\) |
| Conditional expectation | \(E(X)=E(X\mid A)P(A)+E(X\mid A^c)P(A^c)\) |
| Two envelopes | A switching argument that conflicts with symmetry |
For independent \(\operatorname{Gamma}(a,\lambda)\) and \(\operatorname{Gamma}(b,\lambda)\) waiting times, the total is \(\operatorname{Gamma}(a+b,\lambda)\), the fraction of the first is \(\operatorname{Beta}(a,b)\), and the two are independent.
The values of \(X_1,\ldots,X_n\) sorted in increasing order, \(X_{(1)}\le\cdots\le X_{(n)}\), from the minimum to the maximum.
The middle order statistic \(X_{((n+1)/2)}\) for odd \(n\); for even \(n\), commonly the average of the two middle values.
The 75th percentile minus the 25th percentile, a measure of spread built from order statistics.
\(E(X\mid A)\), the expectation of \(X\) computed with probabilities conditional on \(A\).
An argument that always switching between two symmetric envelopes increases the expected amount, contradicting symmetry.
The two-envelope paradox comes from a single conditioning error: plugging in the information \(Y=2X\) and then forgetting that it was given. Conditioning on a pattern’s first tosses shows that a fair coin takes 4 tosses on average to produce HT but 6 to produce HH, because HH overlaps with itself and HT does not. Conditional expectation given an event, \(E(Y\mid X=x)\), is a number that depends on \(x\); replacing \(x\) by \(X\) gives \(E(Y\mid X)\), a random variable that is a function of \(X\). For i.i.d. \(X\) and \(Y\), symmetry and linearity give \(E(X\mid X+Y)=\frac{X+Y}2\). Adam’s law, \(E(E(Y\mid X))=E(Y)\), is the expectation form of the law of total probability.
| Symbol | Meaning |
|---|---|
| \(E(Y\mid A)\) | Expectation of \(Y\) using conditional probabilities given event \(A\) |
| \(g(x)=E(Y\mid X=x)\) | Conditional expectation given the event \(X=x\), a function of \(x\) |
| \(E(Y\mid X)=g(X)\) | Conditional expectation given the random variable \(X\) |
| \(f_{Y\mid X}(y\mid x)\) | Conditional PDF, \(f_{X,Y}(x,y)/f_X(x)\) |
| \(W_{HT}\), \(W_{HH}\) | Number of tosses until the pattern HT or HH first appears |
| \(I\) | Indicator that the second envelope holds the larger amount |
After studying these notes, you should be able to:
Two identical-looking envelopes hold amounts \(X\) and \(Y\), one twice the other. Two arguments conflict.
Argument 1: symmetry. Nothing distinguishes the envelopes, so \(E(Y)=E(X)\).
Argument 2: conditioning. The other envelope holds either twice or half the amount in yours, apparently with probability \(\frac12\) each:
\[E(Y)=E(Y\mid Y=2X)\frac12+E\left(Y\mid Y=\tfrac X2\right)\frac12\overset{?}{=}E(2X)\frac12+E\left(\tfrac X2\right)\frac12=\frac54E(X).\]The two conclusions agree only if \(E(X)\) is 0 or infinite. For positive amounts with a finite mean, one argument must be wrong, and symmetry is hard to dispute.
The flaw. The first line of Argument 2 is correct: it is the law of total probability in expectation form. The error is the next step. Substituting \(Y=2X\) inside \(E(Y\mid Y=2X)\) is fine, but the condition cannot then be dropped:
\[E(Y\mid Y=2X)=E(2X\mid Y=2X),\qquad\text{which need not equal }E(2X).\]A condition can be removed only when what is inside is independent of the condition.
Dependence. Let \(I\) indicate that \(Y=2X\), that is, the second envelope is the larger. Symmetry forces \(E(2X\mid I=1)\ne E(2X)\), so \(X\) and \(I\) are dependent: seeing the amount in your envelope carries information about whether it is the smaller one. That is surprising, since no amount, even a trillion dollars, seems large or small without knowing the scale. But if \(E(X)\) is finite, the distribution of the amounts provides a scale, and the dependence is forced.
Incorrect: Having substituted the given information, the condition can be crossed out.
Correction: Substituting is allowed; forgetting is not. \(E(Y\mid Y=2X)=E(2X\mid Y=2X)\), and the condition may be dropped only if \(X\) is independent of the event \(Y=2X\), which is false here.
A related surprise. If the two envelopes simply hold different unknown positive amounts, with no doubling, there is a strategy that picks the larger one with probability strictly greater than \(\frac12\). Generate your own random threshold \(T\), for example \(T\sim\operatorname{Expo}(1)\), open one envelope, and switch exactly when its amount is below \(T\).
Toss a fair coin repeatedly. Let \(W_{HT}\) and \(W_{HH}\) be the numbers of tosses, including the final two, until HT and HH first appear.
Symmetry gives less than it seems. Swapping the labels heads and tails shows \(E(W_{HH})=E(W_{TT})\) and \(E(W_{HT})=E(W_{TH})\). It says nothing about \(E(W_{HH})\) versus \(E(W_{HT})\), since that comparison would need heads and tails swapped in only one place.
HT by linearity. Wait for the first H; call that time \(W_1\). After that H, the pattern is complete at the first T, since any further H’s leave you still one T away. Call that additional wait \(W_2\). Each wait counts tosses up to and including a success with probability \(\frac12\), so \(W_j-1\sim\operatorname{Geom}(\tfrac12)\) and \(E(W_j)=2\). By linearity,
\[E(W_{HT})=E(W_1)+E(W_2)=2+2=4.\]HH by conditioning. Here a T after the first H destroys the progress. Let \(\mu=E(W_{HH})\) and condition on the first tosses:
Check: \(\frac12(7)+\frac14(2)+\frac14(8)=3.5+0.5+2=6\).
Why the answers differ. Any two fixed positions are equally likely to read HH or HT, each with probability \(\frac14\), so in a long sequence the two patterns occur equally often on average. But HH overlaps with itself: a run HHHHH contains four HH’s. HT cannot overlap with itself this way. So the HH occurrences come in clumps, and since the average number of occurrences is the same, the clumps must be farther apart, which makes the first one later on average. The same question about patterns, called motifs, arises in DNA sequences.
Incorrect: HH and HT are equally likely in any two tosses, so they take equally long to appear.
Correction: Equal frequencies in the long run do not give equal first-appearance times. HH overlaps with itself, so its occurrences cluster, and a tail after an H wastes the progress made.
Given an event. \(E(Y\mid X=x)\) is ordinary expectation computed with the conditional distribution given the event \(X=x\):
\[E(Y\mid X=x)=\sum_yy\,P(Y=y\mid X=x)\quad\text{(discrete)},\qquad E(Y\mid X=x)=\int_{-\infty}^{\infty}y\,f_{Y\mid X}(y\mid x)\,dy\quad\text{(continuous)},\]where \(f_{Y\mid X}(y\mid x)=f_{X,Y}(x,y)/f_X(x)\). The result is a number for each \(x\): write \(g(x)=E(Y\mid X=x)\). It cannot involve \(Y\), which has been averaged over, nor \(X\) as a random variable. If \(X\) and \(Y\) are independent, \(g(x)=E(Y)\), a constant function.
With \(g(x)=E(Y\mid X=x)\),
\[\boxed{E(Y\mid X)=g(X).}\]First compute \(g\) as a function of \(x\), then replace \(x\) by \(X\). If \(g(x)=x^2\), then \(E(Y\mid X)=X^2\). So \(E(Y\mid X)\) is a random variable, and a function of \(X\).
Interpretation. \(E(Y\mid X)\) is the best prediction of \(Y\) when \(X\) is treated as known, best in the sense of minimizing the expected squared error. Any problem written this way can be translated back to conditioning on \(X=x\), which is the thing to do when the notation becomes confusing.
Incorrect: Replace every \(x\) in \(E(Y\mid X=x)\) by \(X\), giving \(E(Y\mid X=X)=E(Y)\).
Correction: The substitution happens after \(g(x)\) has been computed as a function. \(E(Y\mid X)\) means \(g(X)\), not the expectation given the trivially true event \(X=X\).
Let \(X,Y\) be i.i.d. \(\operatorname{Pois}(\lambda)\). Conditional expectations are expectations, so linearity still holds.
Example 1: \(E(X+Y\mid X)\). Treating \(X\) as known:
\[E(X+Y\mid X)=E(X\mid X)+E(Y\mid X)=X+E(Y)=X+\lambda.\]Two general facts were used: a known quantity predicts itself, \(E(h(X)\mid X)=h(X)\) for any function \(h\); and if \(Y\) is independent of \(X\), conditioning on \(X\) does not change it, so \(E(Y\mid X)=E(Y)\). Neither fact used the Poisson.
Example 2: \(E(X\mid X+Y)\). There is no linearity rule for what follows the bar: this is not \(E(X\mid X)+E(X\mid Y)\).
Method 1: the conditional distribution. Let \(T=X+Y\sim\operatorname{Pois}(2\lambda)\). By Bayes’ rule and independence,
\[P(X=k\mid T=n)=\frac{P(Y=n-k)P(X=k)}{P(T=n)} =\frac{\frac{e^{-\lambda}\lambda^{n-k}}{(n-k)!}\cdot\frac{e^{-\lambda}\lambda^k}{k!}}{\frac{e^{-2\lambda}(2\lambda)^n}{n!}} =\binom nk\left(\frac12\right)^n.\]So \(X\mid T=n\sim\operatorname{Bin}(n,\tfrac12)\), with mean \(\frac n2\), and therefore \(E(X\mid T)=\frac T2\).
Method 2: symmetry. Since \(X,Y\) are i.i.d., \(E(X\mid X+Y)=E(Y\mid X+Y)\). Adding and using linearity,
\[2E(X\mid X+Y)=E(X+Y\mid X+Y)=X+Y,\qquad\boxed{E(X\mid X+Y)=\frac{X+Y}2.}\]This argument holds for any i.i.d. pair, not just Poissons. It also matches intuition: if the total of two i.i.d. quantities is 100, the best guess for each is 50.
For any random variables \(X\) and \(Y\) with \(E\lvert Y\rvert<\infty\),
\[\boxed{E\big(E(Y\mid X)\big)=E(Y).}\]\(E(Y\mid X)\) is a random variable, so it has an expectation, and that expectation is the unconditional mean. This is a compact form of the law of total probability: averaging the conditional means over the distribution of \(X\) recovers the overall mean. The proof and many uses come in Lecture 27.
Quick check. In Example 2, \(E\left(\frac{X+Y}2\right)=\frac{2\lambda}2=\lambda=E(X)\).
Try these before reading the answers.
By symmetry between heads and tails, \(E(W_{TH})=E(W_{HT})=4\).
Condition on how the first run of H’s ends. With \(\mu=E(W_{HHH})\): a T first wastes 1 toss; HT wastes 2; HHT wastes 3; HHH finishes in 3.
\[\mu=\frac12(1+\mu)+\frac14(2+\mu)+\frac18(3+\mu)+\frac18\cdot3\quad\Longrightarrow\quad\frac18\mu=\frac74,\quad\mu=14.\]Wait for an H, then for a T: \(E(W_{HT})=\frac1p+\frac1{1-p}\). For \(p=\frac12\) this is 4.
By symmetry all \(E(X_j\mid S_n)\) are equal, and they add to \(E(S_n\mid S_n)=S_n\). So \(E(X_1\mid S_n)=\frac{S_n}n\).
Given \(X\), \(Y\) is Binomial with mean \(10X\), so \(E(Y\mid X)=10X\). By Adam’s law, \(E(Y)=10E(X)=5\).
\(E(S\mid D)=D+E(\text{second die})=D+3.5\), using independence of the dice.
\(\max+\min=X+Y\), so \(E(\max\mid X+Y)=(X+Y)-\frac{X+Y}4=\frac34(X+Y)\).
It replaces \(E(Y\mid Y=2X)\) by \(E(2X)\) and \(E(Y\mid Y=X/2)\) by \(E(X/2)\), dropping the conditions. They cannot be dropped because \(X\) is not independent of which envelope is larger.
\(X\mid X+Y=3\sim\operatorname{Bin}(3,\tfrac12)\), so \(P(X=0\mid X+Y=3)=\frac18\).
| Concept | Essential fact |
|---|---|
| Two-envelope flaw | \(E(Y\mid Y=2X)=E(2X\mid Y=2X)\ne E(2X)\) in general |
| Dropping a condition | Allowed only with independence |
| HT | \(E(W_{HT})=2+2=4\) by linearity |
| HH | \(E(W_{HH})=6\) by conditioning on the first tosses |
| Why they differ | HH overlaps with itself, so its occurrences clump |
| \(E(Y\mid X=x)\) | A number for each \(x\): \(g(x)\) |
| \(E(Y\mid X)\) | The random variable \(g(X)\), a function of \(X\) |
| Known quantities | \(E(h(X)\mid X)=h(X)\) |
| Independence | \(E(Y\mid X)=E(Y)\) |
| Poisson example | \(X\mid X+Y=n\sim\operatorname{Bin}(n,\frac12)\) |
| Symmetry | i.i.d. \(X,Y\): \(E(X\mid X+Y)=\frac{X+Y}2\) |
| Adam’s law | \(E(E(Y\mid X))=E(Y)\) |
An argument that switching between two symmetric envelopes always raises the expected amount. Its flaw is dropping the condition \(Y=2X\) after substituting it.
\(E(Y\mid X=x)\), the expectation of \(Y\) computed with the conditional distribution given \(X=x\); a function of \(x\).
\(E(Y\mid X)=g(X)\), where \(g(x)=E(Y\mid X=x)\). It is a random variable and the best prediction of \(Y\) from \(X\) in mean squared error.
\(E(E(Y\mid X))=E(Y)\), also called iterated expectation or the law of total expectation.
A pattern, such as HH, whose ending can begin another copy of itself. Its occurrences cluster, which lengthens the expected wait for the first one.
Conditional expectation given a random variable obeys a short list of rules: take out what is known, drop what is independent, Adam’s law \(E(E(Y\mid X))=E(Y)\), and the residual \(Y-E(Y\mid X)\) is uncorrelated with every function of \(X\). Geometrically, \(E(Y\mid X)\) is the projection of \(Y\) onto the space of functions of \(X\). Adam’s law is proved by swapping the order of a double sum. Conditional variance is variance computed with everything given \(X\), and Eve’s law splits total variance into the average within-group variance plus the variance between group means: \(\operatorname{Var}(Y)=E(\operatorname{Var}(Y\mid X))+\operatorname{Var}(E(Y\mid X))\). A random sample from a randomly chosen city shows both laws at work.
| Symbol | Meaning |
|---|---|
| \(E(Y\mid X)\) | Best prediction of \(Y\) from \(X\); a function of \(X\) |
| \(h(X)\) | Any function of \(X\) |
| \(Y-E(Y\mid X)\) | Residual: the prediction error |
| \(\operatorname{Var}(Y\mid X)\) | Conditional variance, \(E\big((Y-E(Y\mid X))^2\mid X\big)\) |
| \(Q\sim\operatorname{Beta}(a,b)\) | Random proportion of infected people in a random city |
| \(X\mid Q\sim\operatorname{Bin}(n,Q)\) | Number infected in a sample of \(n\) from that city |
After studying these notes, you should be able to:
A Normal and its square. Let \(Z\sim\mathcal N(0,1)\) and \(Y=Z^2\).
So \(E(Z\mid Z^2)=E(Z)\), even though \(Z\) and \(Z^2\) are dependent. A conditional expectation equal to the unconditional one does not imply independence.
Break a stick of length 1 at a uniformly random point \(X\sim\operatorname{Unif}(0,1)\), keep the left piece of length \(X\), and break it at a uniform point \(Y\):
\[Y\mid X\sim\operatorname{Unif}(0,X).\]This is shorthand for: given \(X=x\), \(Y\sim\operatorname{Unif}(0,x)\). Then \(E(Y\mid X=x)=\frac x2\), so \(E(Y\mid X)=\frac X2\), a random variable. By Adam’s law,
\[E(Y)=E\left(\frac X2\right)=\frac14.\]Linearity also holds: \(E(Y_1+Y_2\mid X)=E(Y_1\mid X)+E(Y_2\mid X)\).
The projection picture. For those who have seen linear algebra: treat random variables with finite variance as vectors, with inner product \(E(UV)\). All functions of \(X\) form a plane through the origin, containing the constants, \(X\), \(X^2\), \(e^X\), and so on. Then \(E(Y\mid X)\) is the projection of \(Y\) onto that plane: the function of \(X\) closest to \(Y\). If \(Y\) is already a function of \(X\), it lies in the plane and projects to itself. The residual is the perpendicular from \(Y\) down to the plane, and property 4 says exactly that it is perpendicular to the plane.
Property 4 from Adam’s law. Expand by linearity:
\[E\Big(\big(Y-E(Y\mid X)\big)h(X)\Big)=E\big(Yh(X)\big)-E\big(E(Y\mid X)h(X)\big).\]In the second term, put back what’s known, \(E(Y\mid X)h(X)=E\big(Yh(X)\mid X\big)\), then apply Adam’s law: it equals \(E(Yh(X))\). The difference is 0. \(\blacksquare\)
Adam’s law, discrete case. Let \(g(x)=E(Y\mid X=x)\), so \(E(Y\mid X)=g(X)\). By LOTUS and the definition of \(g\),
\[E\big(g(X)\big)=\sum_xg(x)P(X=x)=\sum_x\sum_yy\,P(Y=y\mid X=x)P(X=x)=\sum_x\sum_yy\,P(X=x,Y=y).\]Swap the order of summation; \(y\) does not depend on \(x\) and comes out:
\[\sum_yy\sum_xP(X=x,Y=y)=\sum_yy\,P(Y=y)=E(Y).\]Summing the joint PMF over \(x\) gives the marginal of \(Y\). The continuous case is the same with integrals. \(\blacksquare\)
Every expectation is given \(X\), so \(\operatorname{Var}(Y\mid X)\) is a function of \(X\). Omitting the final “\(\mid X\)” in the first form would turn it into a number, which is wrong. The equality of the two forms follows from the properties above, just as for ordinary variance.
Incorrect: \(\operatorname{Var}(Y)=\operatorname{Var}(Y\mid X)\) averaged over \(X\), that is, \(E(\operatorname{Var}(Y\mid X))\).
Correction: That counts only the variability within each value of \(X\). The variability of the conditional means, \(\operatorname{Var}(E(Y\mid X))\), must be added. In the city model, the second term is most of the variance.
The name comes from the order of the operators, E V V E. Adam’s law is E E.
Interpretation. Picture a population split into groups, with \(X\) the group of a randomly chosen person and \(Y\) their height. There are two sources of variability:
Eve’s law says the total variance is simply the sum of the two.
Proof. Let \(g(X)=E(Y\mid X)\). By Adam’s law, \(E(g(X))=E(Y)\) and
\[E\big(\operatorname{Var}(Y\mid X)\big)=E\big(E(Y^2\mid X)-g(X)^2\big)=E(Y^2)-E\big(g(X)^2\big),\] \[\operatorname{Var}\big(g(X)\big)=E\big(g(X)^2\big)-\big(E(Y)\big)^2.\]Adding gives \(E(Y^2)-(EY)^2=\operatorname{Var}(Y)\). \(\blacksquare\)
With \(Y\mid X\sim\operatorname{Unif}(0,X)\), the conditional mean is \(\frac X2\) and the conditional variance is \(\frac{X^2}{12}\). By Eve’s law,
\[\operatorname{Var}(Y)=E\left(\frac{X^2}{12}\right)+\operatorname{Var}\left(\frac X2\right)=\frac1{12}\cdot\frac13+\frac14\cdot\frac1{12}=\frac1{36}+\frac1{48}=\frac7{144}.\]Check directly: \(E(Y^2)=E\left(E(Y^2\mid X)\right)=E\left(\frac{X^2}3\right)=\frac19\), and \(\frac19-\frac1{16}=\frac7{144}\).
To study the prevalence of a disease in a state, choose a city at random, then test a random sample of \(n\) people from that city. Let \(Q\) be the true proportion infected in the chosen city. It is random, since cities differ. Let \(X\) be the number infected in the sample. Model
\[Q\sim\operatorname{Beta}(a,b),\qquad X\mid Q\sim\operatorname{Bin}(n,Q),\]with \(a,b\) known. The Beta is flexible on \((0,1)\) and conjugate to the Binomial. The Binomial assumes sampling with replacement, or a sample small relative to the city.
The mean, by Adam’s law.
\[E(X)=E\big(E(X\mid Q)\big)=E(nQ)=\frac{na}{a+b}.\]The variance, by Eve’s law. Given \(Q\), \(X\) is Binomial, so \(E(X\mid Q)=nQ\) and \(\operatorname{Var}(X\mid Q)=nQ(1-Q)\):
\[\operatorname{Var}(X)=E\big(nQ(1-Q)\big)+\operatorname{Var}(nQ)=n\,E\big(Q(1-Q)\big)+n^2\operatorname{Var}(Q).\]The first term is the within-city variability and the second the between-city variability. Both Beta quantities follow from LOTUS, since multiplying by \(q(1-q)\) keeps the Beta form:
\[E\big(Q(1-Q)\big)=\frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)}\cdot\frac{\Gamma(a+1)\Gamma(b+1)}{\Gamma(a+b+2)}=\frac{ab}{(a+b)(a+b+1)}, \qquad \operatorname{Var}(Q)=\frac{\mu(1-\mu)}{a+b+1},\]where \(\mu=\frac a{a+b}\).
Take \(a=2\), \(b=8\), \(n=50\), so \(\mu=0.2\). Then \(E(X)=10\), and
\[\operatorname{Var}(X)=50\cdot\frac{16}{110}+2500\cdot\frac{0.16}{11}\approx7.27+36.36=43.6.\]A \(\operatorname{Bin}(50,0.2)\) count would have variance 8. Almost all of the extra variance comes from the differences between cities.
Try these before reading the answers.
By symmetry, given \(X^2=a\), \(X=\pm\sqrt a\) equally likely, so \(E(X\mid X^2)=0\). And \(E(X^2\mid X)=X^2\).
\(Y\mid X\sim\operatorname{Unif}(0,1-X)\), so \(E(Y\mid X)=\frac{1-X}2\) and \(E(Y)=\frac14\). \(E(Y^2\mid X)=\frac{(1-X)^2}3\), so \(E(Y^2)=\frac13\cdot\frac13=\frac19\), the same as before by symmetry.
By linearity and Adam’s law, \(E(Y)-E(E(Y\mid X))=E(Y)-E(Y)=0\).
The residual has mean 0, so the covariance is \(E\big(E(Y\mid X)\,(Y-E(Y\mid X))\big)\), which is 0 by property 4 with \(h(X)=E(Y\mid X)\).
\(E(X\mid N)=Np\) and \(\operatorname{Var}(X\mid N)=Np(1-p)\). So \(E(X)=\lambda p\) and \(\operatorname{Var}(X)=E(Np(1-p))+\operatorname{Var}(Np)=\lambda p(1-p)+\lambda p^2=\lambda p\), consistent with \(X\sim\operatorname{Pois}(\lambda p)\) from the chicken-egg story.
Within: \(\frac{25+36+49}3\approx36.7\). Between: the means \(160,170,180\) have variance \(\frac{100+0+100}3\approx66.7\). Total \(\approx103.3\) cm².
\(\mu=\frac12\), \(E(X)=5\). \(E(Q(1-Q))=\frac1{2\cdot3}=\frac16\) and \(\operatorname{Var}(Q)=\frac{1/4}3=\frac1{12}\). So \(\operatorname{Var}(X)=\frac{10}6+\frac{100}{12}=10\). In fact \(X\) is uniform on \(\{0,\ldots,10\}\), whose variance is \(\frac{11^2-1}{12}=10\).
If \(Y=h(X)\), then \(E(Y\mid X)=Y\) and the residual is 0, so \(\operatorname{Var}(Y\mid X)=0\). Eve’s law becomes \(\operatorname{Var}(Y)=0+\operatorname{Var}(Y)\).
\(E(Z\mid Z^2)=0=E(Z)\) for \(Z\sim\mathcal N(0,1)\), yet \(Z^2\) determines \(\lvert Z\rvert\). Equal conditional means say nothing about the rest of the conditional distribution.
| Concept | Essential fact |
|---|---|
| Taking out what’s known | \(E(h(X)Y\mid X)=h(X)E(Y\mid X)\) |
| Dropping what’s independent | \(X,Y\) independent: \(E(Y\mid X)=E(Y)\) |
| Adam’s law | \(E(E(Y\mid X))=E(Y)\) |
| Residual | \(Y-E(Y\mid X)\) is uncorrelated with every \(h(X)\) |
| Projection | \(E(Y\mid X)\) is the closest function of \(X\) to \(Y\) |
| \(E(Z\mid Z^2)\) | 0: magnitude known, sign not |
| Broken stick | \(E(Y)=\frac14\), \(\operatorname{Var}(Y)=\frac7{144}\) |
| Conditional variance | \(E(Y^2\mid X)-(E(Y\mid X))^2\), a function of \(X\) |
| Eve’s law | \(\operatorname{Var}(Y)=E(\operatorname{Var}(Y\mid X))+\operatorname{Var}(E(Y\mid X))\) |
| Within and between | Average within-group variance plus variance of group means |
| City model mean | \(\frac{na}{a+b}\) |
| City model variance | \(nE(Q(1-Q))+n^2\operatorname{Var}(Q)\) |
\(E(h(X)Y\mid X)=h(X)E(Y\mid X)\): a function of the conditioning variable acts as a constant.
\(Y-E(Y\mid X)\), the error in predicting \(Y\) by its conditional expectation. It has mean 0 and is uncorrelated with every function of \(X\).
\(\operatorname{Var}(Y\mid X)=E((Y-E(Y\mid X))^2\mid X)\), the variance of \(Y\) computed with \(X\) treated as known; a function of \(X\).
\(\operatorname{Var}(Y)=E(\operatorname{Var}(Y\mid X))+\operatorname{Var}(E(Y\mid X))\), also called the law of total variance.
A model built in stages, such as a random city proportion \(Q\) followed by a Binomial count given \(Q\). Adam’s and Eve’s laws handle such models by conditioning on the first stage.
A sum of a random number \(N\) of independent terms with mean \(\mu\) and variance \(\sigma^2\), with \(N\) independent of the terms, has mean \(\mu E(N)\) and variance \(\sigma^2E(N)+\mu^2\operatorname{Var}(N)\); conditioning on \(N\) makes both easy. Inequalities give guaranteed bounds rather than approximations, which makes them valuable even when crude. Cauchy–Schwarz bounds \(\lvert E(XY)\rvert\) by marginal second moments and is the statement that correlations lie in \([-1,1]\). Jensen says \(E(g(X))\ge g(E(X))\) for convex \(g\). Markov bounds a tail by the mean, and Chebyshev, which follows from Markov, bounds the chance of being far from the mean by the variance.
| Symbol | Meaning |
|---|---|
| \(N\) | Random number of terms, such as customers in a day |
| \(X_j\) | Amount spent by customer \(j\); mean \(\mu\), variance \(\sigma^2\) |
| \(X=\sum_{j=1}^NX_j\) | Random sum: total spending |
| \(g\) convex | \(g''\ge0\) where it exists; chords lie above the graph |
| \(\mu=E(X)\), \(\sigma=\operatorname{SD}(X)\) | Mean and standard deviation |
| \(a>0\), \(c>0\) | Constants in tail bounds |
After studying these notes, you should be able to:
A store has a random number \(N\) of customers. Customer \(j\) spends \(X_j\), with \(E(X_j)=\mu\) and \(\operatorname{Var}(X_j)=\sigma^2\). Assume \(N,X_1,X_2,\ldots\) are independent: customers do not influence one another, and how many come does not affect how much each spends. Find the mean and variance of the total
\[X=\sum_{j=1}^NX_j.\]A tempting error. Linearity might suggest \(E(X)=N\mu\). That is a category error: the left side is a number and the right side is a random variable. It does suggest the remedy, though: if \(N\) were known, linearity would apply. So condition on \(N\).
The mean. Given \(N=n\), independence of \(N\) and the \(X_j\) lets the condition be dropped, and linearity gives \(E(X\mid N=n)=n\mu\). So \(E(X\mid N)=\mu N\), and by Adam’s law
\[\boxed{E(X)=\mu\,E(N).}\]Written out, this is the law of total probability in expectation form: \(E(X)=\sum_nE(X\mid N=n)P(N=n)=\mu\sum_nnP(N=n)\).
The variance. Given \(N\), the total is a sum of \(N\) independent terms, so \(\operatorname{Var}(X\mid N)=N\sigma^2\). By Eve’s law,
\[\operatorname{Var}(X)=E(N\sigma^2)+\operatorname{Var}(\mu N)=\boxed{\sigma^2E(N)+\mu^2\operatorname{Var}(N).}\]Checks. The mean, average customers times average spending, is intuitive. For units: \(N\) is a count, \(\mu\) is in dollars, and \(\sigma^2\) in square dollars, so both terms of the variance are in square dollars, as they should be. The MGF of \(X\) can be found the same way: given \(N=n\), it is the \(n\)th power of the common MGF of the \(X_j\), and averaging over \(N\) gives the unconditional MGF.
Suppose \(N\sim\operatorname{Pois}(20)\), \(\mu=\$30\), and \(\sigma=\$10\). Then
\[E(X)=30\cdot20=\$600,\qquad\operatorname{Var}(X)=100\cdot20+900\cdot20=20{,}000,\qquad\operatorname{SD}(X)\approx\$141.\]An approximation, such as the Poisson approximation, can be very useful, but “good” has no fixed meaning, and knowing exactly how close it is would amount to knowing the answer. An inequality is a theorem: if a probability is proved to be at most \(0.37\), nothing more needs defending. A bound in one direction is not an estimate, though. A probability at most \(0.38\) could be \(0.004\). Bounds in both directions narrow down the answer. The value of these four inequalities lies in their simplicity and generality, not their sharpness.
For random variables \(X,Y\) with finite variances,
\[\boxed{\lvert E(XY)\rvert\le\sqrt{E(X^2)\,E(Y^2)}.}\]Why it is useful. \(E(XY)\) depends on the joint distribution, through a two-dimensional LOTUS or a transformation. The bound needs only the marginal second moments. If \(X\) and \(Y\) are uncorrelated, \(E(XY)=E(X)E(Y)\) exactly and the bound is not needed.
Statistical meaning. If \(X\) and \(Y\) have mean 0, then \(\operatorname{Corr}(X,Y)=\frac{E(XY)}{\sqrt{E(X^2)E(Y^2)}}\), and Cauchy–Schwarz says exactly that \(\lvert\operatorname{Corr}(X,Y)\rvert\le1\), which was proved in Lecture 21. In linear-algebra terms, \(E(XY)\) plays the role of a dot product.
If \(g\) is convex, then
\[\boxed{E\big(g(X)\big)\ge g\big(E(X)\big).}\]If \(h\) is concave, the inequality reverses: \(E(h(X))\le h(E(X))\).
Convexity. If \(g''\) exists, \(g\) is convex when \(g''(x)\ge0\). In general, \(g\) is convex when every chord between two points of its graph lies on or above the graph, so \(\lvert x\rvert\), with its corner at 0, is convex too. A function \(h\) is concave when \(-h\) is convex, which is why the inequality flips.
Remembering the direction. Use \(g(x)=x^2\): Jensen gives \(E(X^2)\ge(EX)^2\), which is known because variance is nonnegative.
Examples, for \(X>0\):
| Function | Second derivative | Conclusion |
|---|---|---|
| \(1/x\) | \(2/x^3>0\), convex | \(E(1/X)\ge1/E(X)\) |
| \(\log x\) | \(-1/x^2<0\), concave | \(E(\log X)\le\log E(X)\) |
Proof. Let \(\mu=E(X)\). A convex function lies above its tangent line at any point. Take the line \(a+bx\) touching the graph at \((\mu,g(\mu))\), so \(g(x)\ge a+bx\) for every \(x\) and \(a+b\mu=g(\mu)\). This holds for every value of \(X\), so \(g(X)\ge a+bX\) always, and taking expectations,
\[E\big(g(X)\big)\ge a+bE(X)=a+b\mu=g(\mu)=g\big(E(X)\big).\qquad\blacksquare\]For strictly convex \(g\), such as \(x^2\), equality holds only when \(X\) is constant.
Incorrect: \(E(g(X))=g(E(X))\), moving the expectation inside the function.
Correction: That is true for linear \(g\) only. For convex \(g\) the left side is larger, as in \(E(2^X)\ge2^{E(X)}\) for the St. Petersburg game.
For any random variable \(X\) and constant \(a>0\),
\[\boxed{P\big(\lvert X\rvert\ge a\big)\le\frac{E\lvert X\rvert}{a}.}\]Proof. Let \(I\) indicate the event \(\lvert X\rvert\ge a\). Then
\[aI\le\lvert X\rvert\quad\text{always}.\]If \(I=0\), this says \(0\le\lvert X\rvert\). If \(I=1\), it says \(a\le\lvert X\rvert\), which is exactly what \(I=1\) means. Taking expectations and using the fundamental bridge, \(aP(\lvert X\rvert\ge a)\le E\lvert X\rvert\). \(\blacksquare\)
The bound can be useless, for example larger than 1, or infinite if \(E\lvert X\rvert=\infty\), but it is always true.
Intuition with ages. In a group of 100 people, can 95% be younger than the average age? Yes: one extremely old person can pull the mean above almost everyone. Can more than half be older than twice the average age? No: those people alone would already push the total, and so the mean, above its actual value. Likewise at most a third can be older than three times the average. For a nonnegative quantity, Markov’s inequality makes this precise: \(P(X\ge c\,E(X))\le\frac1c\).
If \(X\) has mean \(\mu\) and variance \(\sigma^2\), then for \(a>0\) and \(c>0\),
\[\boxed{P\big(\lvert X-\mu\rvert\ge a\big)\le\frac{\sigma^2}{a^2},\qquad P\big(\lvert X-\mu\rvert\ge c\sigma\big)\le\frac1{c^2}.}\]Proof. Square both sides of the inequality inside the event, then apply Markov to the nonnegative variable \((X-\mu)^2\):
\[P\big(\lvert X-\mu\rvert\ge a\big)=P\big((X-\mu)^2\ge a^2\big)\le\frac{E(X-\mu)^2}{a^2}=\frac{\sigma^2}{a^2}.\]The second form takes \(a=c\sigma\). \(\blacksquare\)
Chebyshev was Markov’s doctoral advisor; Chebyshev’s inequality follows almost immediately from Markov’s.
Crude but universal. Chebyshev guarantees that any distribution with finite variance has at most a \(\frac14\) chance of falling two or more standard deviations from its mean. For a Normal the true chance is about \(0.05\), by the 68-95-99.7% rule. Chebyshev is weaker but needs no assumption about the shape.
Let \(X\sim\operatorname{Expo}(1)\), so \(\mu=\sigma=1\). Bound \(P(X\ge3)\).
Try these before reading the answers.
\(E(X)=\theta\lambda\). With \(\sigma^2=\theta^2\) and \(\operatorname{Var}(N)=\lambda\): \(\operatorname{Var}(X)=\theta^2\lambda+\theta^2\lambda=2\lambda\theta^2\).
With \(E(N)=n\) and \(\operatorname{Var}(N)=0\): mean \(n\mu\) and variance \(n\sigma^2\), as for a sum of \(n\) independent terms.
Apply Cauchy–Schwarz to \(\lvert X\rvert\) and the constant 1: \(E\lvert X\rvert\le\sqrt{E(X^2)\cdot1}\). Squaring gives the result.
\(e^x\) is convex, so \(E(e^X)\ge e^{E(X)}\). \(\sqrt x\) is concave, so \(E(\sqrt X)\le\sqrt{E(X)}\).
\(E(1/X)=\frac12\int_1^3\frac{dx}x=\frac{\log3}2\approx0.549\), and \(1/E(X)=\frac12\). Indeed \(0.549\ge0.5\).
Salaries are nonnegative, so by Markov at most \(\frac{50{,}000}{200{,}000}=\frac14\) of employees earn at least $200,000.
\(P(\lvert X-10\rvert\ge4)\le\frac{4}{16}=\frac14\), so \(P(6<X<14)\ge\frac34\).
If \(P(X=a)=q\) and \(P(X=0)=1-q\), then \(P(X\ge a)=q\) and \(E(X)/a=q\). Markov’s bound cannot be improved in general.
\(\operatorname{Var}(X)=25\), so \(P(\lvert X-50\rvert\ge10)\le\frac{25}{100}=\frac14\). The true value is about \(0.057\).
| Concept | Essential fact |
|---|---|
| Random sum mean | \(E\left(\sum_{j=1}^NX_j\right)=\mu E(N)\) |
| Random sum variance | \(\sigma^2E(N)+\mu^2\operatorname{Var}(N)\) |
| Category error | \(E(X)=N\mu\) mixes a number with a random variable |
| Inequality vs approximation | A bound is a theorem; an approximation needs a notion of “close” |
| Cauchy–Schwarz | \(\lvert E(XY)\rvert\le\sqrt{E(X^2)E(Y^2)}\); correlation in \([-1,1]\) |
| Jensen | Convex \(g\): \(E(g(X))\ge g(E(X))\); concave reverses |
| Direction check | \(E(X^2)\ge(EX)^2\) |
| Markov | \(P(\lvert X\rvert\ge a)\le E\lvert X\rvert/a\) |
| Markov intuition | At most \(1/c\) of a nonnegative population exceeds \(c\) times the mean |
| Chebyshev | \(P(\lvert X-\mu\rvert\ge c\sigma)\le1/c^2\) |
| Two SDs | Chebyshev \(\le0.25\); Normal about \(0.05\) |
A sum \(\sum_{j=1}^NX_j\) whose number of terms \(N\) is itself random.
A function whose chords lie on or above its graph; equivalently \(g''\ge0\) when the second derivative exists.
\(\lvert E(XY)\rvert\le\sqrt{E(X^2)E(Y^2)}\).
\(E(g(X))\ge g(E(X))\) for convex \(g\), and the reverse for concave functions.
\(P(\lvert X\rvert\ge a)\le E\lvert X\rvert/a\) for any \(a>0\).
\(P(\lvert X-\mu\rvert\ge a)\le\sigma^2/a^2\), Markov’s inequality applied to \((X-\mu)^2\).
For i.i.d. random variables with mean \(\mu\) and variance \(\sigma^2\), the two most famous theorems in probability describe the sample mean \(\bar X_n\). The law of large numbers says \(\bar X_n\to\mu\) with probability 1; the weak version follows in one line from Chebyshev. It works by swamping, not by compensating for past luck. The central limit theorem says \(\sqrt n(\bar X_n-\mu)/\sigma\) converges in distribution to \(\mathcal N(0,1)\), whatever the shape of the original distribution: \(\sqrt n\) is exactly the scaling that neither kills nor blows up the error. An MGF calculation with two applications of L’Hôpital’s rule proves it. As a consequence a Binomial is approximately Normal, best when \(p\) is near \(\frac12\), with a continuity correction for discrete values.
| Symbol | Meaning |
|---|---|
| \(X_1,X_2,\ldots\) | i.i.d. with mean \(\mu\) and variance \(\sigma^2<\infty\) |
| \(\bar X_n=\frac1n\sum_{j=1}^nX_j\) | Sample mean of the first \(n\) |
| \(S_n=\sum_{j=1}^nX_j\) | Sum of the first \(n\) |
| \(\xrightarrow{\ d\ }\) | Convergence in distribution: CDFs converge |
| \(\Phi\) | Standard Normal CDF |
| \(M(t)\) | Common MGF of the \(X_j\) |
After studying these notes, you should be able to:
Let \(X_1,X_2,\ldots\) be i.i.d. with mean \(\mu\) and variance \(\sigma^2\), both finite. Observed values become data; with a sample of size \(n\), the natural summary is the sample mean
\[\bar X_n=\frac{X_1+\cdots+X_n}n.\]It is a random variable, while \(\mu\) is a constant. By linearity and independence,
\[E(\bar X_n)=\mu,\qquad\operatorname{Var}(\bar X_n)=\frac{n\sigma^2}{n^2}=\frac{\sigma^2}n.\]Both theorems in this lecture describe \(\bar X_n\) as \(n\) grows.
Meaning. Evaluated at an outcome of the experiment, the random variables \(\bar X_1,\bar X_2,\ldots\) become an ordinary sequence of numbers. The theorem says that, except on an event of probability 0, that sequence converges to \(\mu\).
Coin tossing. With \(X_j\sim\operatorname{Bern}(p)\), \(\bar X_n\) is the proportion of heads in \(n\) tosses, and it converges to \(p\). A million tosses of a fair coin will not give exactly 500,000 heads, but the proportion approaches \(\frac12\). Mathematically, HHHH… forever is a possible sequence, which is why the theorem says “with probability 1”. Without this result, there would be no justification for estimating an unknown \(p\) by a long-run proportion.
Swamping, not compensation. After ten losses in a row, the gambler’s fallacy says a win is “due”. The coin has no memory: the next toss is still fair. The law of large numbers works because any finite run of bad luck, a hundred or a million tosses, is swamped by the infinite future, not because later tosses correct it.
Not a sentence of mediocrity. The theorem assumes i.i.d. observations whose distribution does not change. Someone who trains and improves does not have an i.i.d. sequence of performances. The theorem is what makes learning from data possible: as the sample size grows, the sample mean converges to the truth.
For every \(c>0\),
\[\boxed{P\big(\lvert\bar X_n-\mu\rvert>c\big)\to0\quad\text{as }n\to\infty.}\]Take \(c\) small, such as 0.001: for large \(n\), it is very unlikely that the sample mean is more than 0.001 from \(\mu\). This is convergence in probability. It does not by itself give the pointwise convergence of the strong law, but the intuition is the same.
Proof. By Chebyshev’s inequality,
\[P\big(\lvert\bar X_n-\mu\rvert>c\big)\le\frac{\operatorname{Var}(\bar X_n)}{c^2}=\frac{\sigma^2}{nc^2}\to0.\qquad\blacksquare\]The law of large numbers says \(\bar X_n-\mu\to0\), but not how fast, nor what the distribution of \(\bar X_n\) looks like. To study the rate, multiply by a power of \(n\):
As \(n\to\infty\),
\[\boxed{\sqrt n\,\frac{\bar X_n-\mu}{\sigma}\xrightarrow{\ d\ }\mathcal N(0,1),\qquad\text{equivalently}\qquad\frac{S_n-n\mu}{\sqrt n\,\sigma}\xrightarrow{\ d\ }\mathcal N(0,1).}\]Convergence in distribution means the CDFs converge: the CDF of the left side tends to \(\Phi\) at every point. Unlike the law of large numbers, this is a statement about distributions, not about the random variables converging.
The two forms are the same standardization. \(S_n\) has mean \(n\mu\) and variance \(n\sigma^2\), so subtracting \(n\mu\) (centering) and dividing by \(\sqrt n\sigma\) gives mean 0 and variance 1; dividing numerator and denominator by \(n\) gives the sample-mean form.
Why it is remarkable. The \(X_j\) can be discrete or continuous, symmetric or skewed, nice or ugly; only a finite variance is assumed. Their standardized sum always ends up with the same bell-shaped curve. This is a main reason the Normal distribution is so important, and why Normal approximations for averages are used everywhere: for large \(n\), \(\bar X_n\) is approximately \(\mathcal N(\mu,\sigma^2/n)\).
The theorem holds whenever the variance is finite. Assume in addition that the MGF \(M(t)\) of the \(X_j\) exists, which makes the proof short. The strategy: if MGFs converge to an MGF, the distributions converge to that distribution.
Simplify. Standardizing each term separately, \(Y_j=(X_j-\mu)/\sigma\), gives the same quantity, so assume \(\mu=0\) and \(\sigma=1\). Then the goal is to show that \(S_n/\sqrt n\) has MGF tending to \(e^{t^2/2}\), the \(\mathcal N(0,1)\) MGF.
The MGF of \(S_n/\sqrt n\). By independence the expectation of the product splits, and the terms are identical:
\[E\left(e^{tS_n/\sqrt n}\right)=E\left(e^{tX_1/\sqrt n}\right)\cdots E\left(e^{tX_n/\sqrt n}\right)=M\left(\frac t{\sqrt n}\right)^n.\]Take logs. As \(n\to\infty\), \(M(t/\sqrt n)\to M(0)=1\), so this has the indeterminate form \(1^\infty\). Its log is \(n\log M(t/\sqrt n)=\frac{\log M(t/\sqrt n)}{1/n}\), of the form \(\frac00\). To avoid differentiating with respect to an integer, let \(y=1/\sqrt n\), a real variable tending to 0:
\[\lim_{y\to0}\frac{\log M(yt)}{y^2}.\]Facts about \(M\). \(M(0)=1\), \(M'(0)=E(X)=0\), and \(M''(0)=E(X^2)=1\).
L’Hôpital twice.
\[\lim_{y\to0}\frac{\log M(yt)}{y^2}=\lim_{y\to0}\frac{tM'(yt)}{2yM(yt)}=\frac t2\lim_{y\to0}\frac{M'(yt)}{y}=\frac t2\lim_{y\to0}tM''(yt)=\frac{t^2}2.\]After the first step it is still \(\frac00\), since \(M'(0)=0\), and the factor \(M(yt)\to1\) drops out. Exponentiating, the MGF tends to \(e^{t^2/2}\). \(\blacksquare\)
Historically the first central limit theorem was for the Binomial. Let \(X\sim\operatorname{Bin}(n,p)\). Write \(X=X_1+\cdots+X_n\) with \(X_j\) i.i.d. \(\operatorname{Bern}(p)\), a sum of i.i.d. terms. Standardize with mean \(np\) and standard deviation \(\sqrt{npq}\), \(q=1-p\), and apply the CLT:
How large \(n\) must be has no single answer: a common rule of thumb is about 30, and for the Binomial, \(np\) and \(n(1-p)\) should both be reasonably large.
Normal versus Poisson.
| Poisson approximation | Normal approximation | |
|---|---|---|
| Regime | \(n\) large, \(p\) small, \(\lambda=np\) moderate | \(n\) large, \(p\) not too close to 0 or 1 |
| Works best | Many rare events | \(p\) near \(\frac12\), where the Binomial is nearly symmetric |
The CLT holds for any fixed \(p\), but if \(p=0.001\) the Binomial is very skewed and convergence is slow. The two regimes meet: a Poisson with a large mean is itself approximately Normal.
Continuity correction. A continuous approximation gives \(P(X=a)\approx0\) and ignores whether endpoints are included. Since \(X\) is an integer, \(\{X=a\}=\{a-\frac12<X<a+\frac12\}\), and approximating this interval instead gives a useful answer. In general, widen each end by \(\frac12\):
\[P(a\le X\le b)\approx\Phi\left(\frac{b+\frac12-np}{\sqrt{npq}}\right)-\Phi\left(\frac{a-\frac12-np}{\sqrt{npq}}\right).\]Let \(X\sim\operatorname{Bin}(100,\tfrac12)\), so \(np=50\) and \(\sqrt{npq}=5\). Find \(P(40\le X\le60)\).
For a single value, \(P(X=50)\approx\Phi(0.1)-\Phi(-0.1)\approx0.0797\), against the exact \(\binom{100}{50}2^{-100}\approx0.0796\).
Try these before reading the answers.
By the law of large numbers \(\bar X_n\to\frac12\). By the CLT, \(\bar X_n\) is approximately \(\mathcal N\left(\frac12,\frac1{12n}\right)\).
\(\operatorname{Var}(\bar X_n)=\frac1{4n}\), so \(P(\lvert\bar X_n-\frac12\rvert\ge0.05)\le\frac{1/(4n)}{0.0025}=\frac{100}n\). This is at most 0.05 when \(n\ge2000\).
Need \(0.05\ge1.96\cdot\frac1{2\sqrt n}\), so \(\sqrt n\ge19.6\) and \(n\ge385\). The CLT, using the shape, needs far fewer tosses than Chebyshev’s guarantee.
Each roll has mean 3.5 and variance \(35/12\). The total has mean 2100 and standard deviation \(\sqrt{600\cdot35/12}\approx41.8\). \(P(S>2150)\approx1-\Phi\left(\frac{50}{41.8}\right)\approx1-\Phi(1.20)\approx0.12\).
A \(\operatorname{Pois}(100)\) is a sum of 100 i.i.d. \(\operatorname{Pois}(1)\), so it is approximately \(\mathcal N(100,100)\). \(P(X\ge115)\approx1-\Phi\left(\frac{114.5-100}{10}\right)=1-\Phi(1.45)\approx0.074\).
\(n^{1/4}(\bar X_n-\mu)=n^{-1/4}\cdot\sqrt n(\bar X_n-\mu)\), and the second factor is approximately \(\mathcal N(0,\sigma^2)\), so the product goes to 0. With \(n=\sqrt n\cdot\sqrt n\), the extra \(\sqrt n\) makes the spread grow without bound.
No: the coin is memoryless, and the probability is still \(\frac12\). The law of large numbers holds because the first 10 tosses are swamped by infinitely many later ones, not because heads become more likely.
The CLT requires a finite variance, and the Cauchy has no mean at all. The average of i.i.d. Cauchy variables is again Cauchy, so it never becomes Normal.
\(np=2\) is small, and \(p\) is tiny, so the Binomial is very skewed. The Poisson approximation \(\operatorname{Pois}(2)\) is appropriate.
| Concept | Essential fact |
|---|---|
| Sample mean | \(E(\bar X_n)=\mu\), \(\operatorname{Var}(\bar X_n)=\sigma^2/n\) |
| Strong LLN | \(\bar X_n\to\mu\) with probability 1 |
| Weak LLN | \(P(\lvert\bar X_n-\mu\rvert>c)\to0\); one line from Chebyshev |
| Swamping | Past luck is not corrected, only outweighed |
| CLT, mean form | \(\sqrt n(\bar X_n-\mu)/\sigma\xrightarrow{d}\mathcal N(0,1)\) |
| CLT, sum form | \((S_n-n\mu)/(\sqrt n\sigma)\xrightarrow{d}\mathcal N(0,1)\) |
| Scaling | \(\sqrt n\): larger powers blow up, smaller ones go to 0 |
| Assumption | Finite variance; any shape |
| Proof | \(M(t/\sqrt n)^n\), take logs, L’Hôpital twice, limit \(t^2/2\) |
| Binomial | \(\approx\mathcal N(np,npq)\); best for \(p\) near \(\frac12\) |
| Poisson regime | \(n\) large, \(p\) small |
| Continuity correction | \(P(X=a)=P(a-\frac12<X<a+\frac12)\) |
\(\bar X_n=\frac1n\sum_{j=1}^nX_j\), a random variable with mean \(\mu\) and variance \(\sigma^2/n\) for i.i.d. data.
For i.i.d. variables with finite mean, the sample mean converges to the true mean: with probability 1 (strong law) or in probability (weak law).
A sequence of random variables converges in distribution when their CDFs converge to a limiting CDF.
For i.i.d. variables with finite variance, the standardized sum or sample mean converges in distribution to \(\mathcal N(0,1)\).
Replacing each integer value \(a\) of a discrete variable by the interval \((a-\frac12,a+\frac12)\) before using a continuous approximation.
The belief that independent trials compensate for past outcomes, so that a win becomes “due” after losses.
The last named distributions are all built from Normals. A sum of \(n\) squared i.i.d. standard Normals is Chi-Square with \(n\) degrees of freedom, which is the same as \(\operatorname{Gamma}(n/2,1/2)\). A standard Normal divided by the square root of an independent Chi-Square over its degrees of freedom is Student-t. The t is symmetric, has heavier tails than the Normal, reduces to the Cauchy for one degree of freedom, and approaches the Normal as the degrees of freedom grow, by the law of large numbers. A random vector is Multivariate Normal when every linear combination of its components is Normal. Its MGF follows from the one-dimensional Normal MGF, and within a Multivariate Normal, uncorrelated components are independent.
| Symbol | Meaning |
|---|---|
| \(Z,Z_1,Z_2,\ldots\) | i.i.d. \(\mathcal N(0,1)\) |
| \(V\sim\chi^2_n\) | \(Z_1^2+\cdots+Z_n^2\), Chi-Square with \(n\) degrees of freedom |
| \(T\sim t_n\) | \(Z\big/\sqrt{V/n}\) with \(Z\) independent of \(V\sim\chi^2_n\) |
| \(\mathbf X=(X_1,\ldots,X_k)\) | Random vector |
| \(\mathbf t'\mathbf X=t_1X_1+\cdots+t_kX_k\) | Linear combination of the components |
| \(\mu_j=E(X_j)\) | Mean of component \(j\) |
| MVN | Multivariate Normal |
After studying these notes, you should be able to:
If \(Z_1,\ldots,Z_n\) are i.i.d. \(\mathcal N(0,1)\), then
\[V=Z_1^2+\cdots+Z_n^2\sim\chi^2_n,\]the Chi-Square distribution with \(n\) degrees of freedom.
Sums of squares appear throughout statistics, which is why the Chi-Square, and tests based on it, are used so widely.
A Gamma in disguise. For one degree of freedom, a change of variables shows
\[\chi^2_1=\operatorname{Gamma}\left(\tfrac12,\tfrac12\right).\]Since \(z^2\) is not one-to-one, the derivation goes through the CDF. Lecture 22 found the density \(\frac1{\sqrt{2\pi y}}e^{-y/2}\) for \(y>0\), which is the \(\operatorname{Gamma}(\frac12,\frac12)\) density since \(\Gamma(\frac12)=\sqrt\pi\). Adding \(n\) independent Gammas with the same rate adds their first parameters, so
So the Chi-Square is a special Gamma with its own name, and everything known about the Gamma applies.
Even Normal moments, again. Since \(Z^2\sim\operatorname{Gamma}(\frac12,\frac12)\), the Gamma moment formula, rescaled, gives
\[E(Z^{2n})=E\big((Z^2)^n\big)=2^n\frac{\Gamma(n+\frac12)}{\Gamma(\frac12)}=1\cdot3\cdot5\cdots(2n-1),\]matching the MGF result from Lecture 18: \(E(Z^2)=1\), \(E(Z^4)=3\), \(E(Z^6)=15\).
The t distribution was introduced in 1908 by William Gosset, a statistician working for the Guinness brewery, who published as “Student”, reportedly because Guinness did not want competitors to know it employed a statistician.
If \(Z\sim\mathcal N(0,1)\) and \(V\sim\chi^2_n\) are independent, then
\[\boxed{T=\frac Z{\sqrt{V/n}}\sim t_n,}\]the Student-t distribution with \(n\) degrees of freedom.
The PDF can be found with a Jacobian, but the properties below follow directly from the definition.
Properties.
Approaching the Normal. Construct \(T_n=Z\big/\sqrt{V_n/n}\) with \(V_n=Z_1^2+\cdots+Z_n^2\), using the same \(Z\) and the same sequence \(Z_1,Z_2,\ldots\) for every \(n\); this is allowed, since only the distribution of \(T_n\) matters. Now \(V_n/n\) is the sample mean of i.i.d. \(Z_j^2\), each with mean 1, so by the law of large numbers \(V_n/n\to1\) with probability 1, and so does its square root. Hence \(T_n\to Z\) with probability 1, and
\[\boxed{t_n\xrightarrow{\ d\ }\mathcal N(0,1)\quad\text{as }n\to\infty.}\]For large degrees of freedom, around 30 or more, the t looks very much like a standard Normal.
The Multinomial is the most important discrete multivariate distribution; the Multivariate Normal is the most important continuous one. Independent Normals stacked into a vector are easy. The interesting case allows correlation.
A random vector \(\mathbf X=(X_1,\ldots,X_k)\) is Multivariate Normal if every linear combination
\[t_1X_1+\cdots+t_kX_k\]has a Normal distribution, for every choice of constants \(t_1,\ldots,t_k\). A constant counts as a Normal with variance 0. For \(k=2\) it is called Bivariate Normal.
The definition reduces a vector question back to the one-dimensional Normal.
Example. Let \(Z,W\) be i.i.d. \(\mathcal N(0,1)\). Then \((Z+2W,\ 3Z+5W)\) is Bivariate Normal: for any \(s,t\),
\[s(Z+2W)+t(3Z+5W)=(s+3t)Z+(2s+5t)W,\]a sum of independent Normals, hence Normal. More generally, any vector of linear combinations of i.i.d. Normals is MVN.
Non-example. Let \(Z\sim\mathcal N(0,1)\) and let \(S\) be a random sign, \(\pm1\) with probability \(\frac12\) each, independent of \(Z\). Then \(SZ\sim\mathcal N(0,1)\) by symmetry, so both \(Z\) and \(SZ\) are Normal. But \((Z,SZ)\) is not Bivariate Normal: the combination \(Z+SZ\) equals 0 with probability \(\frac12\), which no Normal does.
The joint MGF. For a random vector, the MGF uses one argument per component:
\[M(\mathbf t)=E\left(e^{\mathbf t'\mathbf X}\right)=E\left(e^{t_1X_1+\cdots+t_kX_k}\right).\]If \(\mathbf X\) is MVN, the exponent \(W=\mathbf t'\mathbf X\) is Normal. A Normal \(W\) has \(E(e^W)=e^{E(W)+\frac12\operatorname{Var}(W)}\), from the univariate MGF \(E(e^{sW})=e^{s\mu+\frac12s^2\sigma^2}\) at \(s=1\). So
where the variance expands into variances and covariances by bilinearity. The joint MGF determines the joint distribution, so an MVN is determined by its means, variances, and covariances.
Split a Multivariate Normal vector as \(\mathbf X=(\mathbf X_1,\mathbf X_2)\). If every component of \(\mathbf X_1\) is uncorrelated with every component of \(\mathbf X_2\), then \(\mathbf X_1\) and \(\mathbf X_2\) are independent.
In general, independent implies uncorrelated but not conversely. Within an MVN the converse also holds; it can be proved by factoring the joint MGF. The non-example shows why the MVN assumption matters: \(Z\) and \(SZ\) are uncorrelated, since \(E(SZ^2)=E(S)E(Z^2)=0\), but clearly dependent.
Let \(X,Y\) be i.i.d. \(\mathcal N(0,1)\). The pair \((X+Y,\ X-Y)\) is Bivariate Normal, since any combination of them is a combination of \(X\) and \(Y\). Their covariance is
\[\operatorname{Cov}(X+Y,X-Y)=\operatorname{Var}(X)-\operatorname{Cov}(X,Y)+\operatorname{Cov}(Y,X)-\operatorname{Var}(Y)=1-1=0.\]So \(X+Y\) and \(X-Y\) are independent. This is special to the Normal: if i.i.d. \(X,Y\) have independent sum and difference, they must be Normal, a much harder theorem.
Incorrect: Two Normal random variables with zero correlation are independent.
Correction: This needs them to be jointly Bivariate Normal. \(Z\) and \(SZ\) are each Normal and uncorrelated, but dependent.
Try these before reading the answers.
\(\operatorname{Gamma}(a,\lambda)\) has mean \(a/\lambda\) and variance \(a/\lambda^2\). With \(a=\frac n2\), \(\lambda=\frac12\): mean \(n\), variance \(2n\).
\(\chi^2_2=\operatorname{Gamma}(1,\tfrac12)=\operatorname{Expo}(\tfrac12)\).
\(E(Z^6)=E((Z^2)^3)\). For \(Y\sim\operatorname{Gamma}(\frac12,\frac12)\), \(E(Y^3)=2^3\frac{\Gamma(\frac72)}{\Gamma(\frac12)}=8\cdot\frac52\cdot\frac32\cdot\frac12=15\).
\(T^2=\frac{Z^2/1}{V/n}\), a ratio of a \(\chi^2_1\) over 1 and an independent \(\chi^2_n\) over \(n\). Such a ratio has an F distribution, here with \((1,n)\) degrees of freedom (supplementary; the F is not covered in this course).
\(E(T)=0\), so \(\operatorname{Var}(T)=E(T^2)=E(Z^2)\,E\left(\frac nV\right)=\frac n{n-2}\). It exceeds 1, reflecting the heavier tails.
Yes: every linear combination of the three is a linear combination of the independent \(Z,W\). \(\operatorname{Cov}(Z+W,Z-2W)=1-2=-1\).
\(\operatorname{Cov}(Z+W,Z+aW)=1+a=0\) when \(a=-1\). The pair is MVN, so \(Z+W\) and \(Z-W\) are then independent.
\(t_1(X+Y)+t_2(X-Y)\) has mean 0 and variance \((t_1+t_2)^2+(t_1-t_2)^2=2t_1^2+2t_2^2\). So \(M(t_1,t_2)=e^{t_1^2}e^{t_2^2}\), a product of two \(\mathcal N(0,2)\) MGFs: the components are independent.
\(t_1\) is Cauchy, and \(E\lvert T\rvert=\infty\). For \(t_3\), \(E\left(1/\sqrt{V/3}\right)<\infty\), so \(E(T)=E(Z)E\left(1/\sqrt{V/3}\right)=0\).
| Concept | Essential fact |
|---|---|
| Chi-Square | \(\chi^2_n=Z_1^2+\cdots+Z_n^2\) |
| Gamma form | \(\chi^2_n=\operatorname{Gamma}(n/2,1/2)\); mean \(n\), variance \(2n\) |
| One degree | \(\chi^2_1=\operatorname{Gamma}(\frac12,\frac12)\) |
| Student-t | \(T=Z/\sqrt{V/n}\), \(Z\) independent of \(V\sim\chi^2_n\) |
| Symmetry | \(-T\) has the same distribution as \(T\) |
| \(t_1\) | Cauchy; no mean |
| Mean | 0 for \(n\ge2\) |
| Tails | Heavier than Normal; moments only below order \(n\) |
| Limit | \(t_n\to\mathcal N(0,1)\) by the law of large numbers |
| MVN | Every linear combination is Normal |
| Non-example | \((Z,SZ)\): each Normal, not jointly |
| MVN MGF | \(\exp(\sum t_j\mu_j+\frac12\operatorname{Var}(\sum t_jX_j))\) |
| Independence | Within an MVN, uncorrelated implies independent |
| Sum and difference | i.i.d. Normals: \(X+Y\) and \(X-Y\) independent |
The distribution of a sum of \(n\) squared i.i.d. standard Normals, equal to \(\operatorname{Gamma}(n/2,1/2)\).
The parameter \(n\) of the Chi-Square and Student-t: the number of squared standard Normals in the Chi-Square.
The distribution of \(Z/\sqrt{V/n}\) with \(Z\sim\mathcal N(0,1)\) independent of \(V\sim\chi^2_n\); symmetric and heavy-tailed, approaching \(\mathcal N(0,1)\) as \(n\) grows.
The distribution of a random vector every linear combination of whose components is Normal.
\(M(\mathbf t)=E(e^{t_1X_1+\cdots+t_kX_k})\), which determines the joint distribution of a random vector.
A Markov chain is a sequence of random states, one step beyond i.i.d.: the next state may depend on the current state, but given the present, the past is irrelevant. Past and future are conditionally independent given the present. A homogeneous chain on finitely many states is described by a transition matrix \(Q\) whose \((i,j)\) entry is the probability of moving from \(i\) to \(j\), with each row summing to 1. If the current distribution is the row vector \(\mathbf s\), then the distribution \(m\) steps later is \(\mathbf sQ^m\), and the \(m\)-step transition probabilities are the entries of \(Q^m\). A distribution with \(\mathbf sQ=\mathbf s\) is stationary: a chain that starts there stays there.
| Symbol | Meaning |
|---|---|
| \(X_0,X_1,X_2,\ldots\) | State of the chain at times \(0,1,2,\ldots\) |
| \(\{1,\ldots,M\}\) | Finite state space |
| \(q_{ij}=P(X_{n+1}=j\mid X_n=i)\) | Transition probability from \(i\) to \(j\) |
| \(Q=(q_{ij})\) | \(M\times M\) transition matrix; rows sum to 1 |
| \(\mathbf s=(s_1,\ldots,s_M)\) | Distribution of a state, written as a row vector |
| \(Q^m\) | \(m\)-step transition matrix |
After studying these notes, you should be able to:
A stochastic process is a collection of random variables evolving over time, such as \(X_0,X_1,X_2,\ldots\), where \(X_n\) is the state of a system at time \(n\). If the \(X_n\) are i.i.d., the process restarts afresh at each step. If they may depend on each other in arbitrary ways, the process can be hopelessly complicated. Markov chains are a compromise, one step beyond i.i.d.
Here time is discrete and the states form a finite set, numbered \(1,\ldots,M\) by convention. Continuous time and continuous or infinite state spaces lead to other processes, studied in a course on stochastic processes.
\(X_0,X_1,X_2,\ldots\) is a Markov chain if, for all \(n\) and all states,
\[\boxed{P(X_{n+1}=j\mid X_n=i,X_{n-1}=i_{n-1},\ldots,X_0=i_0)=P(X_{n+1}=j\mid X_n=i).}\]It is homogeneous when this probability does not depend on \(n\); call it \(q_{ij}\). All chains in these notes are homogeneous.
Interpretation. Think of \(X_n\) as now, \(X_{n+1}\) as tomorrow, and \(X_0,\ldots,X_{n-1}\) as the past. To predict the future, only the current state matters, not the path that led to it. In the language used throughout the course: past and future are conditionally independent given the present. They need not be independent: without knowing \(X_n\), the past can still help predict \(X_{n+1}\), and \(X_{n-1}\) cannot be discarded.
Whether a real system, such as the weather or a stock price, is Markov is an empirical question. The assumption is less restrictive than it looks: a chain whose next state depends on the last two states can be rewritten as an ordinary Markov chain on pairs of states.
A chain can be drawn as states joined by arrows labeled with transition probabilities. A particle wanders from state to state, following arrows.
The transition matrix is \(Q=(q_{ij})\), where row \(i\) lists the probabilities of moving from state \(i\) to each state. For the chain in the figure,
\[Q=\begin{pmatrix} 1/3&2/3&0&0\\ 1/2&0&1/2&0\\ 0&0&0&1\\ 1/2&0&1/4&1/4 \end{pmatrix}.\]Each row sums to 1, since from state \(i\) the chain must go somewhere; columns need not. Conversely, any square matrix with nonnegative entries and row sums 1 is a valid transition matrix. Some books use the transpose, with columns summing to 1.
Incorrect: In a Markov chain, the future is independent of the past.
Correction: The future is independent of the past only given the present. Unconditionally, the past carries information about the present state and hence about the future.
One step. Suppose \(X_n\) has PMF \(\mathbf s\), written as a \(1\times M\) row vector: \(s_i=P(X_n=i)\). By the law of total probability, conditioning on the current state,
\[P(X_{n+1}=j)=\sum_iP(X_{n+1}=j\mid X_n=i)P(X_n=i)=\sum_is_iq_{ij}.\]That sum is the \(j\)th entry of the matrix product \(\mathbf sQ\): a row of \(\mathbf s\) dotted with column \(j\) of \(Q\).
Each step forward in time multiplies on the right by \(Q\).
Two steps. Condition on the missing middle state, then use the Markov property to drop \(X_n\) from the first factor:
\[P(X_{n+2}=j\mid X_n=i)=\sum_kP(X_{n+2}=j\mid X_{n+1}=k)P(X_{n+1}=k\mid X_n=i)=\sum_kq_{ik}q_{kj},\]the \((i,j)\) entry of \(Q^2\). Repeating the argument,
Powers of a single matrix give every multi-step transition probability.
From state 1, what is the chance of being at state 3 two steps later? The paths are \(1\to2\to3\), with probability \(\frac23\cdot\frac12=\frac13\), and no others, since from state 1 the chain goes to 1 or 2 and only state 2 leads to 3. So \((Q^2)_{13}=\frac13\).
Row 1 of \(Q^2\) is \(\left(\frac49,\frac29,\frac13,0\right)\): state 4 cannot be reached from state 1 in two steps.
A probability row vector \(\mathbf s\), with nonnegative entries summing to 1, is a stationary distribution for the chain if
\[\boxed{\mathbf sQ=\mathbf s.}\]Why “stationary”. If \(X_n\) has distribution \(\mathbf s\), then so does \(X_{n+1}\), since \(\mathbf sQ=\mathbf s\), and so does every later state. A chain started at its stationary distribution keeps that distribution forever. For readers who know linear algebra, \(\mathbf s\) is a left eigenvector of \(Q\) with eigenvalue 1, or equivalently an eigenvector of \(Q'\).
Write \(\mathbf s=(s_1,s_2,s_3,s_4)\). The column equations of \(\mathbf sQ=\mathbf s\) are
\[s_2=\tfrac23s_1,\qquad s_3=\tfrac12s_2+\tfrac14s_4,\qquad s_4=s_3+\tfrac14s_4,\]with the first column redundant. The last equation gives \(s_3=\frac34s_4\). Substituting into the middle one, \(\frac34s_4=\frac12s_2+\frac14s_4\), so \(s_2=s_4\). With \(s_2=\frac23s_1\), all are multiples of \(s_1\): \(\mathbf s\propto(1,\frac23,\frac12,\frac23)\propto(6,4,3,4)\). Normalizing,
\[\mathbf s=\left(\frac6{17},\frac4{17},\frac3{17},\frac4{17}\right).\]Check the first column: \(\frac6{17}\cdot\frac13+\frac4{17}\cdot\frac12+\frac4{17}\cdot\frac12=\frac6{17}\).
Questions to answer. Does a stationary distribution exist, as a vector with nonnegative entries? Is it unique? Does the chain converge to it, so that it describes long-run behavior? And how can it be computed efficiently? Solving \(\mathbf sQ=\mathbf s\) directly is a linear system, which can be enormous. Lecture 32 answers these questions.
Two kinds of use.
Markov’s motivation. Markov introduced chains around 1906. The law of large numbers had recently been proved for i.i.d. variables, and some argued that because human behavior is not i.i.d., free will escapes it. Markov answered by going one step beyond i.i.d.: he proved a law of large numbers for chains, showing that independence is not necessary. His first example was a chain on two states, vowel and consonant, estimated from the letters of a Russian novel.
Try these before reading the answers.
Yes. If each \(X_n\) has PMF \(\mathbf p\), the future ignores the present too, and every row of \(Q\) equals \(\mathbf p\).
The past affects the present, which affects the future. Without knowing \(X_n\), the value of \(X_{n-1}\) is informative about \(X_n\) and hence about \(X_{n+1}\). Only once \(X_n\) is known does \(X_{n-1}\) become irrelevant.
With states sunny, rainy: \(Q=\begin{pmatrix}0.8&0.2\\0.4&0.6\end{pmatrix}\).
\((Q^2)_{SR}=0.8\cdot0.2+0.2\cdot0.6=0.28\).
\(s_S=0.8s_S+0.4s_R\) gives \(0.2s_S=0.4s_R\), so \(s_S=2s_R\) and \(\mathbf s=(\frac23,\frac13)\).
\(X_1\): row 1 of \(Q\), \((\frac13,\frac23,0,0)\). \(X_2\): row 1 of \(Q^2\), \((\frac49,\frac29,\frac13,0)\).
\((\frac12,\frac12)Q=(\frac12,\frac12)\). The chain alternates deterministically between the two states. It has a stationary distribution, but started at state 1 its distribution does not converge.
Balance between the states: \(s_1a=s_2b\). So \(\mathbf s=\left(\frac b{a+b},\frac a{a+b}\right)\). Question 5 is the case \(a=0.2\), \(b=0.4\).
By induction, \(\mathbf sQ^m=(\mathbf sQ)Q^{m-1}=\mathbf sQ^{m-1}=\cdots=\mathbf s\).
| Concept | Essential fact |
|---|---|
| Stochastic process | Random variables evolving over time |
| Markov property | Given the present, the past is irrelevant to the future |
| Conditional independence | Past and future are conditionally independent given the present |
| Homogeneous | \(q_{ij}\) does not depend on time |
| Transition matrix | \(Q=(q_{ij})\); nonnegative, rows sum to 1 |
| One step | Distribution \(\mathbf s\mapsto\mathbf sQ\) |
| \(m\) steps | \(\mathbf sQ^m\); \(P(X_{n+m}=j\mid X_n=i)=(Q^m)_{ij}\) |
| Stationary | \(\mathbf sQ=\mathbf s\), \(\mathbf s\) a probability vector |
| Four-state example | \(\mathbf s=(6,4,3,4)/17\) |
| Open questions | Existence, uniqueness, convergence, computation |
| Uses | Modeling and Markov chain Monte Carlo |
| Origin | Markov, about 1906: a law of large numbers beyond i.i.d. |
A collection of random variables indexed by time, describing a system that evolves randomly.
The future is conditionally independent of the past given the present state.
The matrix \(Q\) of one-step transition probabilities \(q_{ij}=P(X_{n+1}=j\mid X_n=i)\), with rows summing to 1.
A probability row vector \(\mathbf s\) with \(\mathbf sQ=\mathbf s\), preserved by every step of the chain.
Computing with a purpose-built Markov chain that converges to a target distribution, using its long-run states as approximate draws.
A chain is irreducible if every state can be reached from every other. A state is recurrent if the chain, started there, returns with probability 1, and transient otherwise; absorbing states trap the chain, and periodic chains cycle. For an irreducible chain on finitely many states, a stationary distribution exists, is unique, and satisfies \(s_i=1/r_i\), where \(r_i\) is the expected return time to \(i\). If some power of \(Q\) has all positive entries, the chain converges to it from any start. Computing it can still be hard. Reversible chains, those with \(s_iq_{ij}=s_jq_{ji}\) for all \(i,j\), make it easy, since such an \(\mathbf s\) is automatically stationary. For a random walk on an undirected network, the stationary probability of each node is proportional to its degree.
| Symbol | Meaning |
|---|---|
| \(Q=(q_{ij})\) | Transition matrix |
| \(\mathbf s\) | Stationary distribution, \(\mathbf sQ=\mathbf s\) |
| \(r_i\) | Expected number of steps to return to \(i\), starting from \(i\) |
| \(\mathbf t\) | Any initial distribution |
| \(s_iq_{ij}=s_jq_{ji}\) | Reversibility condition, also called detailed balance |
| \(d_i\) | Degree of node \(i\): the number of edges at \(i\) |
After studying these notes, you should be able to:
Four small chains show the main behaviors. In each, from a state the chain follows one of its outgoing arrows.
| Chain | Picture | Behavior |
|---|---|---|
| A | States 1, 2, 3; arrows \(1\to1\), \(1\to2\), \(2\to3\), \(3\to1\), \(3\to2\), \(3\to3\) | Every state reachable from every other |
| B | Two copies of chain A on \(\{1,2,3\}\) and \(\{4,5,6\}\), with no arrows between them | Two separate worlds |
| C | States \(0,1,2,3\); from 1 or 2 step left or right; 0 and 3 lead only to themselves | Trapped at an end |
| D | \(1\to2\to3\to1\) with probability 1 | Cycles like a clock |
The four chains.
Recurrence repeats. If a state is recurrent, the chain returns with probability 1; after returning, by the Markov property, the same holds again. So it returns infinitely often. A transient state is visited only finitely many times.
Anything possible eventually happens. In a finite irreducible chain, every state is recurrent: however unlikely a return is on any one excursion, it has positive probability, so it eventually happens.
A transient example. Add one arrow \(3\to6\) to chain B. Then states 1, 2, 3 become transient: the chain may wander among them for a long time, but once it crosses to 6 it can never return.
Let \(X_0,X_1,\ldots\) be an irreducible Markov chain on finitely many states, with transition matrix \(Q\).
The proofs need more linear algebra than these notes cover, but the meaning is clear.
The theorem does not say how to compute \(\mathbf s\). Solving \(\mathbf sQ=\mathbf s\) can be a huge computation, and the return-time formula needs \(r_i\), which is just as hard.
A chain with transition matrix \(Q\) is reversible if there is a probability vector \(\mathbf s\) such that
\[\boxed{s_iq_{ij}=s_jq_{ji}\quad\text{for all states }i,j.}\]Intuition. \(s_iq_{ij}\) is the long-run rate of moves from \(i\) to \(j\). Reversibility says each pair of states exchanges probability equally in both directions. If a chain started at \(\mathbf s\) were filmed and the film played backward, no one could tell the direction of time. In physics this condition is called detailed balance.
If \(s_iq_{ij}=s_jq_{ji}\) for all \(i,j\), then \(\mathbf s\) is stationary.
Proof. Sum both sides over \(i\):
\[\sum_is_iq_{ij}=\sum_is_jq_{ji}=s_j\sum_iq_{ji}=s_j,\]since row \(j\) of \(Q\) sums to 1. The left side is the \(j\)th entry of \(\mathbf sQ\), so \(\mathbf sQ=\mathbf s\). \(\blacksquare\)
Finding an \(\mathbf s\) that satisfies the reversibility equations takes insight rather than routine elimination, but when it can be done, no matrix equations need solving, and the answer often works for chains of any size.
An undirected network has nodes joined by edges that work in both directions. A random walk on it moves, at each step, along one of the edges at the current node, chosen uniformly at random. Assume the network is connected, so the walk is irreducible, and has no loops from a node to itself. Let \(d_i\) be the degree of node \(i\), the number of edges at \(i\).
Claim. \(d_iq_{ij}=d_jq_{ji}\) for all \(i,j\).
Normalizing gives a reversible, hence stationary, distribution:
The walk spends time at each node in proportion to the node’s number of connections. The argument works for a network with four nodes or four billion.
Nodes 1, 2, 3, 4 with edges \(\{1,2\}\), \(\{1,3\}\), \(\{2,3\}\), \(\{3,4\}\). The degrees are \(d=(2,2,3,1)\), summing to 8, so
\[\mathbf s=\left(\frac28,\frac28,\frac38,\frac18\right).\]The expected return time to node 4 is \(r_4=1/s_4=8\) steps.
Incorrect: Every Markov chain is reversible with respect to its stationary distribution.
Correction: Every irreducible finite chain has a stationary distribution, but most chains are not reversible. Chain D, the clock, has stationary distribution \((\frac13,\frac13,\frac13)\), yet \(s_1q_{12}=\frac13\) while \(s_2q_{21}=0\). Run backward, the clock would visibly turn the other way.
Try these before reading the answers.
States 0 and \(N\) are absorbing, hence recurrent. States \(1,\ldots,N-1\) are transient, since with positive probability the chain is absorbed and never returns. The chain is reducible.
In a finite chain some state is visited infinitely often. By irreducibility, every state can be reached from it with positive probability, and repeated chances make each state visited infinitely often. So all states are recurrent.
\(\mathbf s=(\frac13,\frac13,\frac13)\), and each return takes exactly 3 steps, matching \(1/s_i=3\). From \((1,0,0)\), \(\mathbf tQ^n\) cycles through \((1,0,0)\), \((0,1,0)\), \((0,0,1)\) and does not converge; no power of \(Q\) is all positive.
The only condition with \(i\ne j\) is \(s_1a=s_2b\). With \(s_1+s_2=1\): \(\mathbf s=\left(\frac b{a+b},\frac a{a+b}\right)\). Every two-state chain is reversible.
The center has degree \(k\) and each outer node degree 1, total \(2k\). So \(s_{\text{center}}=\frac12\), each outer node has \(\frac1{2k}\), and the expected return time to the center is 2, as it must be: every walk alternates between center and outside.
All degrees are 2, so \(\mathbf s=(\frac14,\frac14,\frac14,\frac14)\). The walk alternates between the two pairs of opposite corners, so it is periodic and does not converge from a fixed start.
Degrees: 4 corners of degree 2, 4 edge midpoints of degree 3, 1 center of degree 4; total \(8+12+4=24\). A corner has \(s=\frac2{24}=\frac1{12}\), so the expected return time is 12.
With \(s_i=\frac1M\) for all \(i\), \(s_iq_{ij}=\frac{q_{ij}}M=\frac{q_{ji}}M=s_jq_{ji}\). So the uniform distribution is reversible, hence stationary.
The chain is reducible. Point mass at either absorbing state is stationary, and so is any mixture of the two; uniqueness requires irreducibility.
| Concept | Essential fact |
|---|---|
| Irreducible | Every state reachable from every state |
| Recurrent | Return with probability 1, hence infinitely often |
| Transient | Positive probability of never returning |
| Absorbing | Once entered, never left |
| Periodic | Cycles deterministically in time; blocks convergence |
| Finite irreducible | All states recurrent |
| Existence and uniqueness | Finite irreducible chains have exactly one stationary \(\mathbf s\) |
| Return time | \(s_i=1/r_i\) |
| Convergence | If some \(Q^m>0\): \(\mathbf tQ^n\to\mathbf s\) for every start |
| Interpretation | \(s_i\) is the long-run fraction of time at \(i\) |
| Reversible | \(s_iq_{ij}=s_jq_{ji}\) for all \(i,j\) |
| Reversible implies stationary | Sum over \(i\) |
| Undirected network | \(s_i\propto d_i\) |
A chain in which every state can be reached from every other state with positive probability in some number of steps.
A state is recurrent if the chain started there returns with probability 1, and transient otherwise.
A state with \(q_{ii}=1\), which the chain never leaves once it arrives.
A chain whose returns to a state can occur only at multiples of some integer greater than 1, preventing convergence of its distribution.
The condition \(s_iq_{ij}=s_jq_{ji}\) for all \(i,j\), also called detailed balance; it implies that \(\mathbf s\) is stationary.
The number of edges at a node of a network. A random walk on an undirected network has stationary probabilities proportional to degree.
Put symmetric weights \(w_{ij}=w_{ji}\) on the edges of an undirected network and move from \(i\) to \(j\) with probability proportional to \(w_{ij}\). This weighted random walk is reversible, and its stationary probability at \(i\) is proportional to the total weight \(\sum_kw_{ik}\) at \(i\), the weighted degree. Conversely, every reversible chain is such a walk, with \(w_{ij}=s_iq_{ij}\). Google PageRank is a non-reversible example: a page’s importance is the stationary probability of a random web surfer who follows links, jumps to a random page from pages without links, and with probability 0.15 teleports to a random page at each step. Teleportation makes the chain irreducible with all transition probabilities positive, so PageRank exists, is unique, and can be computed by running the chain, using the sparsity of the link matrix.
| Symbol | Meaning |
|---|---|
| \(w_{ij}=w_{ji}\ge0\) | Edge weight; 0 when there is no edge |
| \(\sum_kw_{ik}\) | Weighted degree of node \(i\) |
| \(Q\) | Link-following matrix of the web |
| \(J\) | \(M\times M\) matrix of all 1’s |
| \(G=\alpha Q+(1-\alpha)J/M\) | Google matrix, with \(\alpha=0.85\) originally |
| \(\mathbf s\) | PageRank vector: stationary distribution of \(G\) |
After studying these notes, you should be able to:
Start with an undirected network, possibly with loops, and put a weight \(w_{ij}\ge0\) on each edge, with \(w_{ij}=0\) when there is no edge. Because edges work in both directions,
\[w_{ij}=w_{ji}.\]The weighted random walk moves from \(i\) to \(j\) with probability proportional to \(w_{ij}\):
\[q_{ij}=\frac{w_{ij}}{\sum_kw_{ik}},\]assuming every node has some positive weight. With all weights equal to 1, this is the unweighted walk of Lecture 32, where \(\sum_kw_{ik}=d_i\). A loop at a node counts once toward that node’s degree.
Reversibility. Multiply \(q_{ij}\) by its denominator and use symmetry:
\[\left(\sum_kw_{ik}\right)q_{ij}=w_{ij}=w_{ji}=\left(\sum_kw_{jk}\right)q_{ji}.\]This is the reversibility condition with \(s_i\) proportional to the weighted degree:
Nodes 1, 2, 3 with \(w_{12}=1\), \(w_{13}=2\), \(w_{23}=3\). The weighted degrees are \(1+2=3\), \(1+3=4\), and \(2+3=5\), totaling 12, so
\[\mathbf s=\left(\frac3{12},\frac4{12},\frac5{12}\right).\]Check one reversibility equation: \(s_1q_{13}=\frac3{12}\cdot\frac23=\frac16\) and \(s_3q_{31}=\frac5{12}\cdot\frac25=\frac16\).
Any reversible Markov chain is a random walk on some weighted undirected network.
Proof. Let the chain have transition matrix \(Q\) and satisfy \(s_iq_{ij}=s_jq_{ji}\), with all \(s_i>0\); states with \(s_i=0\) can be removed. Use the states as nodes and define
\[w_{ij}=s_iq_{ij}.\]By reversibility, \(w_{ij}=s_jq_{ji}=w_{ji}\), so the weights are symmetric. The weighted walk on this network moves from \(i\) to \(j\) with probability
\[\frac{w_{ij}}{\sum_kw_{ik}}=\frac{s_iq_{ij}}{s_i\sum_kq_{ik}}=\frac{q_{ij}}1=q_{ij},\]since row \(i\) of \(Q\) sums to 1. This is exactly the original chain. \(\blacksquare\)
So weighted random walks on undirected networks are, in a sense, the entire theory of reversible chains. The difficulty for a given chain is finding the \(s_i\) in the first place. Non-reversible chains correspond to directed networks, where weights may differ in the two directions, and are much harder.
A search engine must order the pages that match a query: with millions of matches, which comes first? Early approaches used human curation, or counted how often a page used the query word, which is easy to abuse and a poor measure of quality. Later engines used the link structure: a page is important if many pages link to it. That too can be gamed, with thousands of dummy pages, and a link from an unimportant page should count for little.
The insight of PageRank, from Sergey Brin and Larry Page in 1998: a page is important if important pages link to it. That definition looks circular, but it is an equation. Let \(s_j\) be the score of page \(j\). Page \(i\) divides its score equally among its outgoing links, since a page that links to a thousand others recommends each one less. So
\[s_j=\sum_is_iq_{ij},\qquad q_{ij}=\frac1{\#\text{links on page }i}\ \text{if }i\text{ links to }j.\]A page with no links would make its row of \(Q\) all zeros; replace that row by \(1/M\) in every entry, as if the page linked to everything.
Page 1 links to 2 and 3; page 2 links to 1 and 3; page 3 links to 4; page 4 has no links. Then
\[Q=\begin{pmatrix} 0&1/2&1/2&0\\ 1/2&0&1/2&0\\ 0&0&0&1\\ 1/4&1/4&1/4&1/4 \end{pmatrix}.\]The random surfer. \(Q\) is the transition matrix of a person surfing the web at random: on each page, click a random link; on a page with no links, open a random page. The score equation \(\mathbf s=\mathbf sQ\) says that \(\mathbf s\), once normalized, is the stationary distribution of this chain. A page’s importance is the long-run fraction of time the random surfer spends there.
For the four-page web, solving \(\mathbf sQ=\mathbf s\) gives \(\mathbf s=\left(\frac2{11},\frac2{11},\frac3{11},\frac4{11}\right)\). Page 4 ranks highest: everything reaching page 3 flows on to it.
This chain is not reversible: page 3 links to 4, and page 4 sends only a quarter of its weight back. So the stationary distribution cannot be read off as in Part 1.
Incorrect: PageRank ranks pages by how many links point to them.
Correction: Each incoming link is weighted by the score of the page it comes from, divided among that page’s outgoing links. Page 4 has a single incoming link but ranks highest, because page 3 passes it all of its score.
On the real web it is unclear whether every page can be reached from every other by following links, and convergence could be slow. Brin and Page’s paper used instead
where \(J\) is the matrix of all 1’s, \(M\) the number of pages, and \(0<\alpha<1\); the original paper suggested \(\alpha=0.85\). \(J/M\) is itself a valid transition matrix: jump to a uniformly random page.
Interpretation. At each step the surfer flips a coin: with probability \(\alpha\), follow the link structure, using \(Q\); with probability \(1-\alpha\), teleport to a uniformly random page.
Why teleport. Every entry of \(G\) is at least \((1-\alpha)/M>0\). Though tiny when \(M\) is in the billions, it is positive. So \(G\) is irreducible with \(G^1\) all positive, and the main theorem of Lecture 32 applies: the stationary distribution, the PageRank vector, exists, is unique, and the chain converges to it from any start. The chain is still not reversible.
For the four-page web with \(\alpha=0.85\), the PageRank vector is approximately \((0.192,\,0.192,\,0.273,\,0.343)\). Teleportation pulls the scores slightly toward uniform.
Solving \(\mathbf sG=\mathbf s\) by Gaussian elimination takes on the order of \(M^3\) operations. With \(M=10^{10}\) pages, that is about \(10^{30}\), impossible even on fast computers. Instead, use convergence: start from any initial distribution \(\mathbf t\), such as uniform or all mass on page 1, and compute
\[\mathbf tG,\quad\mathbf tG^2,\quad\mathbf tG^3,\ \ldots,\]each from the previous by one multiplication, until the vector stops changing.
Each step is cheap.
\[\mathbf tG=\alpha\,\mathbf tQ+(1-\alpha)\frac{\mathbf tJ}M.\]How many steps are needed is a hard question for a chain this large; in practice the iteration runs until the vector appears to have stabilized, which gives an approximate PageRank.
Core idea: The Markov chain view gives PageRank both its meaning, the long-run time a random surfer spends on each page, and a practical way to compute it, by simulating the chain’s distribution forward.
Try these before reading the answers.
Weighted degrees: \(1,\ 3,\ 3,\ 1\), total 8. \(\mathbf s=(\frac18,\frac38,\frac38,\frac18)\).
Node 1’s weighted degree becomes 3, total 10: \(\mathbf s=(\frac3{10},\frac3{10},\frac3{10},\frac1{10})\).
With \(\mathbf s=(\frac b{a+b},\frac a{a+b})\): \(w_{11}=s_1(1-a)\), \(w_{22}=s_2(1-b)\), and \(w_{12}=w_{21}=\frac{ab}{a+b}\).
Column 1: \(\frac2{11}\cdot\frac12+\frac4{11}\cdot\frac14=\frac2{11}\). Column 2: the same, \(\frac2{11}\). Column 3: \(\frac2{11}\cdot\frac12+\frac2{11}\cdot\frac12+\frac4{11}\cdot\frac14=\frac3{11}\). Column 4: \(\frac3{11}+\frac4{11}\cdot\frac14=\frac4{11}\).
\(0.85\,(\frac12,0,\frac12,0)+0.0375\,(1,1,1,1)=(0.4625,\ 0.0375,\ 0.4625,\ 0.0375)\).
Every entry of \(J/M\) is \(\frac1M\ge0\) and each row sums to 1. The \(j\)th entry of \(\mathbf tJ\) is \(\sum_it_i=1\), so \(\mathbf tJ/M=(\frac1M,\ldots,\frac1M)\).
As \(\alpha\to0\), the surfer always teleports and PageRank becomes uniform, ignoring links. As \(\alpha\to1\), it approaches the stationary distribution of \(Q\) alone, which may fail to exist uniquely or converge slowly on the real web.
Dummy pages that nobody links to receive little PageRank themselves, roughly only the teleportation share, so their recommendations carry little weight.
Under \(G\), \(s_3g_{34}\) involves \(g_{34}=0.85+0.0375=0.8875\), while \(s_4g_{43}=s_4\cdot0.25\). With \(s_3\approx0.273\) and \(s_4\approx0.343\): \(0.242\ne0.086\).
| Concept | Essential fact |
|---|---|
| Weighted walk | \(q_{ij}=w_{ij}/\sum_kw_{ik}\), with \(w_{ij}=w_{ji}\) |
| Stationary | \(s_i\propto\sum_kw_{ik}\) |
| Unweighted case | \(s_i\propto d_i\) |
| Reversible chains | All are weighted walks, with \(w_{ij}=s_iq_{ij}\) |
| PageRank idea | Important pages are linked to by important pages |
| Score equation | \(\mathbf s=\mathbf sQ\): stationary distribution of the random surfer |
| Dangling page | Row replaced by \(1/M\) |
| Google matrix | \(G=\alpha Q+(1-\alpha)J/M\), \(\alpha=0.85\) |
| Teleportation | Makes every entry positive: existence, uniqueness, convergence |
| Not reversible | Links are one-way |
| Computation | Iterate \(\mathbf t\mapsto\mathbf tG\); \(Q\) sparse, \(\mathbf tJ\) trivial |
| Four-page example | About \((0.19,0.19,0.27,0.34)\) |
A walk on a network with symmetric edge weights that moves along each edge with probability proportional to its weight.
The total weight \(\sum_kw_{ik}\) of the edges at node \(i\); the stationary probability of a weighted walk is proportional to it.
The stationary distribution of the Google chain, used to rank web pages by the long-run fraction of time a random surfer spends on them.
Jumping to a uniformly random page, with probability \(1-\alpha\) at each step, which makes the Google chain irreducible with all transition probabilities positive.
A matrix with mostly zero entries, such as the web’s link matrix, which can be stored and multiplied efficiently.
Ten ideas carry most of the course: conditioning, symmetry, random variables and their distributions, stories, linearity, indicator random variables, LOTUS, the law of large numbers, the central limit theorem, and Markov chains. They group into three themes: how to think about randomness, how to compute expectations, and long-run behavior. Two applications show the tools beyond probability. In linear regression \(Y=\beta_0+\beta_1X+\varepsilon\) with \(E(\varepsilon\mid X)=0\), taking covariances and using Adam’s law gives the slope \(\beta_1=\operatorname{Cov}(X,Y)/\operatorname{Var}(X)\). In survey sampling, dividing each observation by its probability of being sampled gives the unbiased Horvitz–Thompson estimator of a population total, proved with indicators and linearity. Basu’s elephant shows that unbiasedness alone does not make an estimator good.
| Symbol | Meaning |
|---|---|
| \(Y=\beta_0+\beta_1X+\varepsilon\) | Simple linear regression model |
| \(\beta_0,\beta_1\) | Unknown intercept and slope |
| \(\varepsilon\) | Error term, with \(E(\varepsilon\mid X)=0\) |
| \(y_1,\ldots,y_N\) | Fixed values for a finite population of size \(N\) |
| \(p_j\) | Known probability that unit \(j\) is included in the sample |
| \(I_j\) | Indicator that unit \(j\) is in the sample |
| \(\hat\tau\) | Horvitz–Thompson estimator of \(\tau=\sum_jy_j\) |
After studying these notes, you should be able to:
| Theme | Idea | What it does |
|---|---|---|
| Thinking about randomness | 1. Conditioning | Conditional probability and expectation, Bayes’ rule, conditional independence, Markov chains |
| 2. Symmetry | Collapses long calculations to one line, but must actually be present | |
| 3. Random variables and their distributions | The objects of study, kept distinct from each other | |
| 4. Stories | Explain why the named distributions matter and how they connect | |
| Computing expectations | 5. Linearity | Holds with or without independence |
| 6. Indicator random variables | Turn counts into sums and expectations into probabilities | |
| 7. LOTUS | Expectations of functions without finding their distributions | |
| Long-run behavior | 8. Law of large numbers | Sample means converge to the true mean |
| 9. Central limit theorem | Standardized sums become Normal | |
| 10. Markov chains | Long-run behavior one step beyond i.i.d. |
Expectations reach beyond averages: variances, covariances, and probabilities (by the fundamental bridge) are all expectations.
Probability and inference. Probability starts from a model, such as \(\operatorname{Pois}(\lambda)\) with \(\lambda\) known, and asks what data are likely. Statistical inference goes the other way: from observed data, estimate unknown parameters or predict future data. Inference needs probability for both its language and its theorems.
Simple linear regression models how \(X\) predicts \(Y\):
\[Y=\beta_0+\beta_1X+\varepsilon,\qquad E(\varepsilon\mid X)=0.\]The error \(\varepsilon\) is often assumed Normal, but only the conditional mean assumption is needed here: no value of \(X\) makes the errors systematically positive or negative.
Step 1: the error has mean 0. By Adam’s law, \(E(\varepsilon)=E\big(E(\varepsilon\mid X)\big)=0\).
Step 2: the error is uncorrelated with \(X\). Since \(E(\varepsilon)=0\), \(\operatorname{Cov}(\varepsilon,X)=E(\varepsilon X)\). Condition on \(X\) and take out what’s known:
\[E(\varepsilon X)=E\big(E(\varepsilon X\mid X)\big)=E\big(X\,E(\varepsilon\mid X)\big)=E(X\cdot0)=0.\]Step 3: take the covariance of both sides with \(X\). The constant drops out, and by bilinearity,
\[\operatorname{Cov}(Y,X)=\beta_1\operatorname{Var}(X)+\operatorname{Cov}(\varepsilon,X)=\beta_1\operatorname{Var}(X).\]The intercept follows by taking expectations of both sides of the model. These are the population versions. The familiar sample formulas, with sums of \((x_i-\bar x)(y_i-\bar y)\), estimate them, and much of regression and econometrics rests on conditional expectation, Adam’s law, and Eve’s law, often viewed geometrically as projection.
A finite population has \(N\) units, such as people, each with a fixed but unknown value \(y_j\): a height, an income, an opinion. Here the \(y_j\) are constants, not random. The goal is the total \(\tau=\sum_{j=1}^Ny_j\), or equivalently the mean \(\tau/N\).
A sample of size \(n\) is drawn by some scheme in which unit \(j\) is included with known probability \(p_j>0\). In a simple random sample all \(p_j\) are equal, but in practice some people are easier to reach than others. The randomness comes only from the sampling. Let \(I_j\) indicate that unit \(j\) is in the sample.
Divide each sampled value by its inclusion probability and add:
\[\boxed{\hat\tau=\sum_{j\in\text{sample}}\frac{y_j}{p_j}=\sum_{j=1}^N\frac{y_j}{p_j}I_j.}\]If the sampled values are 5, 10, and 15 with inclusion probabilities \(a\), \(b\), \(c\), the estimate is \(\frac5a+\frac{10}b+\frac{15}c\).
Unbiasedness. Written over the sample, the denominators are random, since which units appear is random. Written with indicators over the whole population, the denominators are constants, and linearity with the fundamental bridge finishes the proof:
\[E(\hat\tau)=\sum_{j=1}^N\frac{y_j}{p_j}E(I_j)=\sum_{j=1}^N\frac{y_j}{p_j}p_j=\sum_{j=1}^Ny_j=\tau.\qquad\blacksquare\]This idea, also called inverse probability weighting, is used very widely. A unit that was unlikely to be sampled stands in for many similar units, so its value is weighted up.
A circus owner wants the total weight of 50 elephants. Weighing elephants is hard, so he proposes to weigh one typical elephant, Stampy, and multiply by 50. A statistician objects that this is biased and insists on the Horvitz–Thompson estimator. The owner wants to weigh Stampy, and the method allows any positive inclusion probabilities, so they choose Stampy with probability \(0.99\) and each of the other 49 elephants with probability \(\frac{0.01}{49}=\frac1{4900}\).
They draw Stampy, as expected. The Horvitz–Thompson estimate is
\[\hat\tau=\frac{\text{Stampy's weight}}{0.99}\approx1.01\times\text{Stampy's weight},\]about the weight of a single elephant, as an estimate for fifty. Had another elephant been drawn, the estimate would have been 4900 times its weight. Averaged over the sampling, the estimator is unbiased, yet in every outcome it is absurd.
Incorrect: An unbiased estimator is a good estimator.
Correction: Unbiasedness is about the average over hypothetical repetitions. An estimator can be unbiased and still be wildly wrong every time, with enormous variance. What makes an estimator good, and how frequentist and Bayesian approaches answer that, are central questions of statistical inference.
Try these before reading the answers.
Take expectations: \(E(Y)=\beta_0+\beta_1E(X)+E(\varepsilon)\), and \(E(\varepsilon)=0\).
\(\operatorname{Cov}(X,Y)=0.5\cdot2\cdot3=3\), so \(\beta_1=\frac34\).
\(\beta_1=\frac{\operatorname{Corr}(X,Y)\operatorname{SD}(X)\operatorname{SD}(Y)}{\operatorname{SD}(X)^2}=\operatorname{Corr}(X,Y)\frac{\operatorname{SD}(Y)}{\operatorname{SD}(X)}\).
\(E(Y\mid X)=\beta_0+\beta_1X\) and \(\operatorname{Var}(Y\mid X)=\operatorname{Var}(\varepsilon\mid X)\). Eve’s law gives \(\operatorname{Var}(Y)=E(\operatorname{Var}(\varepsilon\mid X))+\beta_1^2\operatorname{Var}(X)\): unexplained plus explained variance.
\(\hat\tau=\frac Nn\sum_{j\in\text{sample}}y_j\), and dividing by \(N\) gives the sample mean.
Estimates: \(\frac2{0.5}=4\), \(\frac4{0.3}\approx13.33\), \(\frac6{0.2}=30\). Mean: \(0.5(4)+0.3(13.33)+0.2(30)=2+4+6=12=\tau\).
\(E(\hat\tau^2)=0.5(16)+0.3(177.8)+0.2(900)\approx241.3\), so \(\operatorname{Var}\approx241.3-144=97.3\). With uniform \(p_j=\frac13\), the estimates are \(6,12,18\), with variance \(24\). Probabilities that ran opposite to the sizes of the \(y_j\) made the estimator much worse.
With probability 0.99 the estimate is \(w/0.99\); with probability 0.01 it is \(4900w\). Mean \(=w+49w=50w\). Unbiased, but the estimate is either about \(w\) or \(4900w\), never close to \(50w\).
Indicator random variables, linearity, and the fundamental bridge, which connects expectations of indicators to probabilities.
| Concept | Essential fact |
|---|---|
| Ten ideas | Conditioning, symmetry, random variables and distributions, stories, linearity, indicators, LOTUS, LLN, CLT, Markov chains |
| Three themes | Randomness; computing expectations; long-run behavior |
| Inference | From data to unknown parameters and predictions |
| Regression model | \(Y=\beta_0+\beta_1X+\varepsilon\), \(E(\varepsilon\mid X)=0\) |
| Key step | \(\operatorname{Cov}(\varepsilon,X)=E(X\,E(\varepsilon\mid X))=0\) |
| Slope | \(\beta_1=\operatorname{Cov}(X,Y)/\operatorname{Var}(X)\) |
| Intercept | \(\beta_0=E(Y)-\beta_1E(X)\) |
| Horvitz–Thompson | \(\hat\tau=\sum y_j/p_j\) over the sample |
| Unbiasedness proof | Rewrite with indicators; linearity; \(E(I_j)=p_j\) |
| Basu’s elephant | Unbiased yet absurd; unbiasedness is not enough |
Using observed data to estimate unknown parameters or predict future observations, the reverse direction from probability.
A model \(Y=\beta_0+\beta_1X+\varepsilon\) with \(E(\varepsilon\mid X)=0\), so that \(E(Y\mid X)\) is linear in \(X\).
An estimator whose expected value equals the quantity it estimates.
The estimator \(\sum y_j/p_j\) of a population total, summing each sampled value divided by its inclusion probability; also called inverse probability weighting.
A story showing that an unbiased estimator can give absurd estimates in every outcome, so unbiasedness alone does not make an estimator good.