Problem

Suppose a coin has head probability $p$.

(a) If we toss the coin n times, where n is a fixed constant, and let $X, Y$ denote the number of heads and tails, respectively. Show that $X$ and $Y$ are dependent.

(b) If we toss the coin $N$ times, where $N \sim \text{Poisson}(\lambda)$ and let $X, Y$ denote the number of heads and tails, respectively. Show that $X$ and $Y$ are independent. What are their distributions?

Solution

(a)

First, let's note that $X \sim \text{Binomial}(n, p)$ and $Y \sim \text{Binomial}(n, 1-p)$

To check whether $X$ and $Y$ are independent, we can see whether $f_{X,Y}$ equals $f_X \times f_Y$.

$f_{X,Y}(x,y) = \text{Multinomial}(n, (p, 1-p)) = \frac{n!}{x!y!} p^x (1-p)^y$

whereas

$f_X \times f_Y = \frac{n!}{(n-x)!x!} \frac{n!}{(n-y)!y!} p^x(1-p)^{n-x} p^{n-y}(1-p)^{n-y}$

These clearly are not equal, so $X$ and $Y$ are dependent.

(b)

I was stumped by this. My thought was we could view this problem as a transformation of random variables, transforming a Poisson r.v by a transformation function equal to the PMF of a Binomial, but this didn't lead anywhere.

Deriving $X$ and $Y$

Looking at the solution key, the correct approach to find the distribution for $X$ when $n$ is not fixed, but draws from $N \sim \text{Poisson}(\lambda)$ is:

$P(X=x) = \sum_{n=x}^{\infty} P(X=x \vert N = n) P(N=n)$

I think this derives from

$P(X=x) = \sum_{n=x}^{\infty} P(X=x, N = n)$

where $P(X=x, N = n) = P(X=x \vert N = n) P(N=n)$ by conditional probability.

going back to

$P(X=x) = \sum_{n=x}^{\infty} P(X=x, N = n)$

why is this true? In this case, we are using the marginal mass formula; if $(X,Y)$ have a joint distribution then the marginal mass function for $X$ is defined by

$f_X(x) = P(X=x) = \sum_n P(X=x, N=n) = \sum_n f(x,n)$

that is, if you sum over all values of $N$, you are back to just $X$.

That leads me to another question about this problem: in summing over all values for $N$, why do we sum from x to $\infty$ instead of from 0 to $\infty$? $X$ is the number of heads given we flip a coin $N$ times, I suppose it is because we must have flipped the coin at least $x$ times in order to have yielded $x$ heads? I can buy it, but I'm still a tad frustrated that I don't think I could have come up with this myself.

So to summarize, to correctly frame this problem, the insights are:

  • Define $P(X=x)$ in terms of its marginal distribution over the joint distribution with $N$
  • Use conditional probability to avoid needing the joint distribution function

And once the problem is correctly framed, it is a matter of applying the Binomial and Poisson probability mass functions:

$P(X=x) = \sum_{n=x}^{\infty} P(X=x \vert N = n) P(N=n) \\ = \sum_{n=x}^{\infty} \binom{n}{x} p^x(1-p)^{n-x}\dfrac{e^{-\lambda}\lambda^n}{n!}$

and algebraic manipulation to

$= e^{-\lambda} \sum_{n=x}^{\infty} \dfrac{1}{x!(n-x)!}(p\lambda)^x ((1-p)\lambda)^{n-x} $

and a cute substitution, let $m = n -x$

$P(X=x) \\ = \dfrac {e^{-\lambda}(p\lambda)^x}{x!} \sum_{m=0}^{\infty} \frac{1}{m!}((1-p)\lambda)^m \\ = \dfrac {e^{-\lambda}(p\lambda)^x}{x!} e^{(1-p)\lambda} \\ = \dfrac{e^{-p\lambda}(p\lambda)^x}{x!} $

so we find that $P(X=x) = \text{Poisson}(p\lambda)$.

By similar reasoning, we get $P(Y=y) = \text{Poisson}((1-p)\lambda)$.

Showing $X$ and $Y$ are independent.

To show $X$ and $Y$ are independent, we need to show that the joint density $P(X=x, Y=y)$ is equal to the product of the individual density functions $P(X=x) \times P(Y=y)$.

Let's start by expressing the joint density using the law of total probability over the partition of every value $n$ can take:

$P(X=x, Y=y) = \sum_{n=0}^{\infty} P(X=x, Y=y \vert N=n)P(N=n)$

let's consider just $P(X=x, Y=y \vert N=n)$. Given $N=n$, we are left with a Binomial. There is also the fact that for any pair $(x,y)$, only one conditional ($N=n=x+y$) can contribute mass. This means $P(X=x, Y=y \vert N=n) = \binom{n}{x} p^x (1-p)^{n-x} \text{Indicator}(x+y)$.

Plugging this back into the sum we have:

$P(X=x, Y=y) = \sum_{n=0}^{\infty} \binom{n}{x} p^x (1-p)^{n-x} \text{Indicator}(x+y) \dfrac{e^{-\lambda}\lambda^n}{n!}$

Since this expression only has value for $n=x+y$ we plug in $x+y$ for $n$ and drop the summation, leaving:

$\binom{x+y}{x} p^x (1-p)^{(x+y)-x} \dfrac{e^{-\lambda}\lambda^{x+y}}{(x+y)!} \\ = \dfrac{1}{x!y!}p^x(1-p)^y e^{-\lambda} \lambda^{x+y} \\ = \dfrac{(p\lambda)^x e^{-p\lambda}}{x!} \dfrac{((1-p)\lambda)^x e^{-(1-p)\lambda}}{y!} \\ = P(X=x) P(Y=y)$

Sources

  • 36-700 hw3 problem 4