The inverse problem

forward problem

Model + parameters known → predict data. Easy ("two lines of Python" for a fair coin).

inverse problem

Data (effect) known → infer model/cause. Hard: different causes explain the same data.

a theory earns credibility

by surviving independent datasets across scales.

Three types of inverse problem

1 · hypothesis testing

Is a statement consistent with data? ("markets always crash")

2 · parameter estimation

What value does θ take? (TED spread threshold before 2008)

3 · model selection

Which of several fitting models? Hardest — needs the evidence term.

Where uncertainty comes from

• Noise in the measurement
• Intrinsic randomness (no two petals alike)
• Designed randomness of complex systems (lottery drum)
• Epistemic stance: causes unknown (age of universe ± error). Aleatory: games of chance, "40% rain tomorrow".

The calculus of probability

bounds · sum rule

$0\le P\le1$, $\;\sum_i P(i)=1$, $\;\int P(x)\,dx=1$

product rule

$P(X,Y)=P(X\mid Y)P(Y)=P(Y\mid X)P(X)$

independent

$P(X,Y)=P(X)P(Y)$ — only then multiply marginals

mutually exclusive

$P(x_1\cup\dots\cup x_n)=\sum P(x_i)$

Total probability → Bayes

total probability

$P(E)=\sum_i P(E\mid A_i)P(A_i)$

Bayes (two hypotheses)

$$P(H\mid E)=\frac{P(E\mid H)P(H)}{P(E\mid H)P(H)+P(E\mid H^c)P(H^c)}$$

the maths is trivial

It's a rearranged product rule. Bayes' contribution was the interpretation: a way to update belief.

Inference form

$P(\theta\mid D,I,M)=\dfrac{P(D\mid\theta,I,M)\,P(\theta\mid I,M)}{P(D\mid I,M)}$

posterior

belief about θ after the data — what you want

likelihood

P(data | θ); product over independent data points only (apples from different, randomly chosen trees)

prior

everything known before the experiment

evidence

$\int P(D\mid\theta)P(\theta)\,d\theta$ — all ways the data could arise

Working rules

• Posterior ∝ likelihood × prior for estimation and testing. Need the evidence only for model selection.
• θ is meaningless without its model $M$; I and M are always silently conditioned on.
• Flat prior $P(\theta)=$ const when you know nothing.

Updating

$P(\theta\mid D_1,D_2)\propto P(D_2\mid\theta)\,P(\theta\mid D_1)$

Yesterday's posterior is today's prior. Old data is never thrown away (needs $D_1, D_2$ independent).

prior-sensitivity test

If a different reasonable prior changes the answer a lot → you need more data.

Traps

• $P(C\mid G)+P(C\mid G^c)\ne1$ in general. Only $P(C\mid G)+P(C^c\mid G)=1$.
• Dropping implicit conditioning ("you chose door $a$", "a theft did happen").
• Rule 1: write down exactly what you want to know. "There are no rules for $n>1$."

Population vs sample

population

infinite, holds the truth; never accessible

sample

finite $n$ draws; gives estimates (finite sums), not calculations (integrals)

statistic

any summary of a finite sample, no model assumed: mean, variance, skewness, kurtosis, correlation

Histogram → PDF

• Equal-width bins; height = count in the bin ∝ probability density.
• Smooth the bin tops → approximate PDF ("a poor person's PDF").
• Hard edges, empty bins; converges to the true PDF as $n$ grows.
• Try several bin widths — the apparent shape changes.

Density, joint, marginal

density

probability mass per unit length/area/volume, e.g. $\dfrac{dB}{dH\,dW}$

"average density"

$=1/V$ — meaningless; only local density informs

marginalise

$P(X)=\int P(X,Y)\,dY$ — integrate out what you don't want

joint needed when

the observable depends on several variables (BMI from $h,w$)

Definitions

random variable

$X:\Omega\to\mathbb{R}$; support $S=X(\Omega)$

PMF (discrete)

$p(x)\ge0$, $\sum_{S}p(x)=1$ — values are probabilities

PDF (continuous)

$f\ge0$, $\int f=1$ — not a probability; can exceed 1 ($f=2$ on $(0,\tfrac12)$)

link

$F(x)=\int_{-\infty}^x f(t)\,dt$, $\;f=F'$ (ignore finitely many kinks; set 0 there)

CDF $F(x)=P(X\le x)$

• Defined on all of ℝ: write it piecewise, never as a 3-column table.
• Valid CDF ⇔ (1) non-decreasing, (2) right-continuous $F(a)=F(a^+)$, (3) $F(-\infty)=0$, $F(\infty)=1$.
• Unknown $k$: impose right-continuity for candidates, then non-decreasing to reject the wrong root.
• $P(a<X\le b)=F(b)-F(a)$.

Jumps classify $X$

jump size = point mass

$P(X=a)=F(a)-F(a^-)$, $\;F(a^-)=\lim_{h\to0}F(a-h)$

Σ jumps = 1

discrete

no jumps

continuous: $P(X=a)=0$, so $<$ vs $\le$ doesn't matter

0 < Σ jumps < 1

mixed (rest spread over an interval)

Expectation

$E(X)$

$\sum x\,p(x)$ or $\int x\,f(x)\,dx$

exists only if

$E|X|<\infty$ — check first (modulus on $x$, never on $f$)

$E[g(X)]$

$\sum g(x)p(x)$ — swap $x$ for $g(x)$, keep the density

Gamma shortcut

$\int_0^\infty x^{\alpha-1}e^{-x}dx=\Gamma(\alpha)=(\alpha-1)!$

odd / even

odd over symmetric range → 0; even → 2× half

Moments

raw (about 0)

$\mu_r'=E(X^r)$; $\mu_1'$ = mean

central (about mean)

$\mu_r=E[(X-\mu_1')^r]$; $\mu_1=0$ always

variance

$\mu_2=\mu_2'-(\mu_1')^2=E[X^2]-(E X)^2$; $=0$ ⇔ degenerate

skewness

$\gamma_1=\mu_3/\mu_2^{3/2}$ — keeps the sign

kurtosis

$\beta_2=\mu_4/\mu_2^2$, $\gamma_2=\beta_2-3$ (normal: $\beta_2=3$)

Reading $\gamma_1$ and $\gamma_2$

• $\gamma_1>0$: long right tail (positive skew). $\gamma_1<0$: long left tail. $=0$: symmetric.
• $\beta_1=\mu_3^2/\mu_2^3\ge0$ loses the sign — prefer $\gamma_1$.
• $\gamma_2>0$ leptokurtic (more peaked), $=0$ mesokurtic, $<0$ platykurtic.
• Unit-free: cm → m scales $\mu_3$ by $10^{-6}$; ratios cancel it.

The limit chain

Binomial → Poisson

$n\to\infty$, $p\to0$, $np=\lambda$ finite ($x\ll n$)

Poisson → Normal

also $\lambda\gg1$, look near the mean, and treat the variable as continuous

Poisson error bar

var = mean, so a count $n$ is quoted $n\pm\sqrt n$ (5 accidents/h → $5\pm\sqrt5$)

Shapes

• Binomial $\gamma_1=\frac{q-p}{\sqrt{npq}}$: $p<\tfrac12$ right-tailed, $p=\tfrac12$ symmetric.
• Binomial $\gamma_2=\frac{1-6pq}{npq}$: leptokurtic for $p<0.2113$ or $p>0.7887$, platykurtic between.
• Poisson: small λ piles at 0; larger λ → symmetric.
• Exponential: $\gamma_1=2$, $\gamma_2=6$. Uniform: symmetric, platykurtic, $\mu_4=\frac{(b-a)^4}{80}$.
• Gamma: $\gamma_1=\frac2{\sqrt\alpha}$, $\gamma_2=\frac6\alpha$, so exponential's 2 and 6 at $\alpha=1$, heading to normal as $\alpha$ grows.

Memoryless + simulation

geometric

$P(X\ge a)=q^a$; $P(X\ge k+t\mid X\ge k)=q^t=P(X\ge t)$

exponential

$P(X>d+x\mid X>d)=e^{-\lambda x}=P(X>x)$

only ones

geometric (discrete) and exponential (continuous)

prob. integral transform

$F_X(X)\sim U(0,1)$ → simulate $x_i=F^{-1}(u_i)$

Normal toolkit

standardise

$Z=\dfrac{X-\mu}{\sigma}\sim N(0,1)$

interval

$P(a\le X\le b)=\Phi\!\left(\frac{b-\mu}\sigma\right)-\Phi\!\left(\frac{a-\mu}\sigma\right)$

symmetry

$\Phi(0)=\tfrac12$, $\;\Phi(-z)=1-\Phi(z)$ — one table side is enough

no closed-form CDF

hence tables; read them backwards too ($\Phi(z)=0.75$ → $z\approx0.675$ by interpolation)

Area rule

$\mu\pm1\sigma$

68.27% (34.1% each side)

$\mu\pm2\sigma$

95.45% (+13.6% each side)

$\mu\pm3\sigma$

99.73% — only ~27 in 10,000 fall outside

moments

odd central = 0; $\mu_4=3\sigma^4$; $\mu_{2n}=1\cdot3\cdots(2n-1)\sigma^{2n}$

Central Limit Theorem

• Sum/mean of many i.i.d. draws from any parent with finite variance → Gaussian. Infinite variance ("pathological") → no CLT.
• Sample size 1 = no summing = no CLT (you just get the parent back).
• Bigger sample size → narrower and faster; skewed parents converge slower.
• Learn the implications; the derivation isn't examined.

Markov

$X\ge0$, $a>0$

$P(X\ge a)\le\dfrac{E X}{a}$

generalised

any $X$, $g\ge0$: $P(g(X)\ge a)\le\dfrac{E[g(X)]}{a}$

proof idea

drop $\int_0^a$, then $x\ge a$ on $[a,\infty)$

Chebyshev

upper

$P(|X-\mu|\ge\varepsilon)\le\dfrac{\sigma^2}{\varepsilon^2}$

lower

$P(|X-\mu|<\varepsilon)\ge1-\dfrac{\sigma^2}{\varepsilon^2}$

kσ form

$\le1/k^2$: at least 75% within 2σ for any distribution (normal: 95.45%)

from Markov

$g=(x-\mu)^2$, $a=\varepsilon^2$

Convexity and Jensen

test

$f''\ge0$ convex (chord above), $f''\le0$ concave; $f$ convex ⇔ $-f$ concave

Jensen

convex: $E[h(X)]\ge h(EX)$; concave: $\le$

uses

$x^2$ → $E X^2\ge(EX)^2$ so Var ≥ 0; $\log$ → $E\log X\le\log EX$

Joint and marginal PMF

joint PMF

$p(x,y)=P(X=x,Y=y)\ge0$, $\;\sum_x\sum_y p(x,y)=1$

marginals

$p_X(x)=\sum_y p(x,y)$ (row sums), $\;p_Y(y)=\sum_x p(x,y)$ (column sums)

dependent ranges

swapping the order keeps the answer but changes the limits: $\sum_{y=0}^{2}\sum_{x=0}^{y}=\sum_{x=0}^{2}\sum_{y=x}^{2}$

Joint CDF

definition

$F(x,y)=P(X\le x,\,Y\le y)$: the lower-left quadrant

properties 1–4

non-decreasing and right-continuous in each coordinate; $\to0$ as $x\to-\infty$ or $y\to-\infty$; $\to1$ as both $\to\infty$

5 · rectangle inequality

$F(x_2,y_2)-F(x_2,y_1)-F(x_1,y_2)+F(x_1,y_1)\ge0$. Not implied by 1–4.

marginal CDF

$F_X(x)=\lim_{y\to\infty}F(x,y)$: send the other variable to $+\infty$

Joint PDF

definition

$F(x,y)=\int_{-\infty}^{x}\int_{-\infty}^{y}f(u,v)\,dv\,du$; $\;f\ge0$, $\iint f=1$

back to PDF

$f=\dfrac{\partial^2F}{\partial x\,\partial y}$ where $f$ is continuous

regions

$P((X,Y)\in A)=\iint_A f\,dy\,dx$; $\;f_X(x)=\int f(x,y)\,dy$

uniform on a region

$f=\dfrac1{\text{area}}$, so probability $=$ area ratio

Conditional distribution

why a limit

$P(Y=y)=0$ for continuous $Y$, so condition on $y-\varepsilon<Y\le y+\varepsilon$ and let $\varepsilon\to0^+$

result

$f_{X\mid Y}(x\mid y)=\dfrac{f(x,y)}{f_Y(y)}$, needs $f_Y(y)>0$. For fixed $y$ it is a PDF in $x$.

multiplication rule

$f(x,y)=f_{Y\mid X}(y\mid x)\,f_X(x)=f_{X\mid Y}(x\mid y)\,f_Y(y)$

conditional mean

$E[Y\mid X=x]=\int y\,f_{Y\mid X}(y\mid x)\,dy$

Independence

definition

$F(x,y)=F_X(x)F_Y(y)$ for all $x,y$

any one of

$p=p_Xp_Y$ · $f=f_Xf_Y$ · $M(t_1,t_2)=M_X(t_1)M_Y(t_2)$ near $(0,0)$

factorisation

$f=g(x)\,h(y)$ is enough; $g,h$ need not be densities

support test

rectangular support is necessary, not sufficient. Triangle or disk ⇒ dependent.

dependence

one point with $p(x,y)\ne p_X(x)p_Y(y)$ settles it

Covariance

expectation

$E[g(X,Y)]=\sum\sum g\,p$ or $\iint g\,f$; $\;\mu'_{r,s}=E[X^rY^s]$

formula

$\operatorname{Cov}(X,Y)=E[XY]-E[X]E[Y]$; $\;\operatorname{Cov}(X,X)=\operatorname{Var}X$

linear

$\operatorname{Cov}(aX+b,cY+d)=ac\operatorname{Cov}(X,Y)$, bilinear in each slot

variance of a sum

$\operatorname{Var}(aX+bY)=a^2\operatorname{Var}X+b^2\operatorname{Var}Y+2ab\operatorname{Cov}$

one-way street

independent ⇒ Cov $=0$. Cov $=0$ ⇏ independent ($X\sim U(-1,1)$, $Y=X^2$).