Category Archives: Neyman: From Life

Constance Reid, Neyman: From Life

Standard boring academic cover, with basically just the words on a yellowy cover Jerzy Neyman is one of those epochal researchers who developed tools that working scientists use every day, without necessarily knowing that they’re doing so. In 200 years they’ll still be using his tools, and he’ll have descended even more into the obscurity of constant use: mathematicians don’t need to write “quadratic reciprocity (Gauss, 1801)”; likewise, even when people have forgotten that it’s the Neyman-Pearson Lemma, they’ll still be using the Neyman-Pearson Lemma.

What this famous lemma gives us is a rigorous reason to use the particular statistical methods that we’d been using on intuitive grounds previously. It shows that in a certain well-defined sense, the best statistical test we can use is one based on likelihood ratios. A likelihood ratio, in turn, is the ratio of two probabilities: the probability that the data you see before you would have arisen under one hypothesis, divided by the probability that it would have arisen under another hypothesis. [1]

Neyman’s work laid the foundations of mathematical statistics, in the sense of deriving statistical results with the rigor of mathematicians. The decades following Neyman and Pearson’s work saw probability theory become rigorously grounded in measure theory, and saw statistics rigorously grounded in probability theory. And it saw statistical methods applied to countless problems well beyond gambling and agriculture. Neyman was at the center of all of this: he started what quickly became the world’s greatest center of statistical research, at Berkeley, and applied rigorous methods all the way from the foundations of statistics up to the structure of galaxies. He was, by any measure, an awe-inspiring scientist.

Constance Reid managed to catch Neyman while he was still alive. She interviewed him every Saturday toward the end of his life, and published the book soon after he died. Her biography isn’t scholarly, in the sense of digging very far into the content of his work. I think it’s fairer to call it an “academic” biography: it spends most of its time following Neyman’s university career, the conferences he organized, and the spats he got into with other statistical luminaries — principally the legendary R.A. Fisher, whose immortality in genetics is just as assured as it is in statistics. Fisher doesn’t come out looking very good under Reid’s lens: he’s bitter, condescending, egomaniacal, imperious, and unwilling to brook even the slightest disagreement on his work.

Reid has a bit of a fine line to toe: go into too much detail about the content of Neyman’s work and she’s likely to alienate general readers; go into too little detail and she’ll alienate technical readers. (Neyman: From Life is published by Springer-Verlag, the canonical publisher of math textbooks, so it’s clear that she has a technical audience in mind.) I think she stays too far on the non-technical side: we learn a lot about the conferences Neyman traveled to and the bureaucratic dust-ups he started at Berkeley, but very little about what, exactly, got Fisher and him so angry at each other. Was it that Fisher came from an earlier era when intuitive derivations of statistical results were acceptable? Was it that Fisher found Neyman’s methods appropriate for large samples but useless for small ones? Did it have something to do with “fiducial inference,” which is a Fisher innovation that I’ve never seen anyone explain clearly? (Check out its Wikipedia entry and try to explain to me what “fiducial inference” even means.) It’s not always clear that Reid herself knows the answers to these questions, and from time to time she describes Neyman glossing over some details during their interviews; he may not have thought much of her technical chops either.

That’s really a small gripe, of course: most people really will not need the kind of intellectual biography that I hoped to find. Instead they’ll meet a statistician who’s still unfairly productive in his 80s, whom the university can’t possibly let go of without forfeiting hundreds of thousands of dollars in grant money and a world-renowned conference every year. [2] Neyman seems like a perfectly lovable mensch, ambling about with his cigarettes, prodding students into doing math at the blackboard, and traveling everywhere with his “long-time collaborator and constant companion”, Professor Betty Scott.

(Reid’s coverage of Neyman’s private life is hilariously restrained. It would seem, on the basis of a few hints from others throughout the book, that Neyman was something of a ladies’ man. Reid seems to have shied away from asking Neyman’s wife what she thought about this; either that, or she avoided writing any of it down.)

Reid’s book has certainly rekindled my love for mathematical statistics, which I enjoyed studying so much at Carnegie Mellon. I’m definitely going to revisit the subject now that I’m older and maybe marginally smarter.

[1] — Here I simply must link to Cosma Shalizi’s clever derivation of the Neyman-Pearson Lemma on economic grounds.

[2] — The full text of the Proceedings of the Berkeley Symposia on Mathematical Statistics and Probability is available online? That’s several thousand pages. It contains many groundbreaking papers, among them the one in which Charles Stein introduced what we now call Stein’s Paradox. Amazing.