Hypothesis Testing
Hypothesis testing in ML
Where all of this actually shows up once you're building models instead of running trials.
TODO chapter. Rough plan for where hypothesis testing quietly runs through machine learning practice:
- A/B testing and online experimentation, the multiple testing problem in the wild.
- Model comparison and whether a benchmark improvement is actually significant.
- Coverage guarantees with a testing flavor, tying back to conformal-style methods.
- Where the classical machinery starts to strain under high-dimensional, non-iid, or adaptively collected data, and what people do about it.