While we’re talking about Fourier series

(…as we were), does anyone have an intuition — or can anyone point me to an intuition — for why Fourier series would be so much more powerful than power series? Intuitively, I would think that very-high-order polynomials would buy you the power to represent very spiky functions, functions with discontinuities at a point (e.g., f(x) = -1 for x less than 0, f(x) = 1 for x >= 0), etc. Yet the functions that can be represented by power series are very smooth (“analytic“), whereas the functions representable by Fourier series can be very spiky indeed.

The intuition may be in Körner, but I’ve not found it.

This could lead me down a generalization path, namely: develop a hierarchy of series representations, with representations higher on the hierarchy being those that can represent all the functions that those lower on the hierarchy can represent, plus others. In this way you’d get a total ordering of the set of series representations. I don’t know if this is even possible; maybe there are some series representations that intersect with, but are not sub- or supersets of, other series representations. I don’t think I’ve ever read a book that treated series representations generally; it’s always been either Fourier or power series, but rarely both, and never any others. Surely these books exist; I just don’t know them.

And now, back to reading Hawkins.

My dime-store understanding of measure theory and its history

I’m really enjoying Thomas Hawkins’s Lebesgue’s Theory of Integration: Its Origins and Development. It’s a historical treatment of where measure theory, and the modern theory of integration (in the calculus sense) came from. I’m coming at this without knowing much of the mathematics, apart from a general outline. That makes some of the reading unclear, but I’m getting it.

The basic thrust seems to start with Fourier, or maybe there is a parallel track starting with Cauchy and Riemann. Fourier comes up with the idea of representing a function as an infinite sum of sines and cosines, which immediately brings out a bunch of mathematical puzzles. In particular, when are you allowed to integrate a Fourier series term by term? That is, when is the integral of the sum equal to the sum of the integrals? While this may not seem like a practical question, it very much is. I can testify to this in my limited capacity as an amateur mathematician: you want to be able to perform operations on symbols without thinking terribly hard about it. It would be nice if you could just say “the integral of the sum is the sum of the integrals” without thinking. And, long story short, it turns out that you can say that (or so I gather) if you’re talking about an integral in the sense of Lebesgue rather than an integral in the sense of Riemann.

It takes a while to get there, though. And when Riemann introduces his definition of the integral, which is applicable to a wide swath of functions, many (all?) mathematicians believed that the integral concept had reached its “outermost limits” (to quote Paul du Bois-Reymond). It took half a century and more of mathematicians studying the structure of the real numbers, teasing out the fine distinctions between different subtle classes of real numbers, before we arrived at a theory of integration that handled all of these cases correctly. Now we can talk coherently about the integral of a function which takes value 1 for every rational number and takes value 0 for every irrational number.

Tracing the path from Riemann to Lebesgue is fascinating, for at least a couple reasons. First, I think it conflicts with an idealized picture of mathematicians carefully progressing from one obviously true statement to another via the ineluctable laws of logic. As Hawkins writes of Hermann Hankel’s purported proof that a nowhere-dense set can be covered by sets of zero content, “Here Hankel’s actual understanding — as opposed to his formal definition — of a ‘scattered’ set becomes more evident.” For decades, mathematicians didn’t have a stock of counterexamples ready to hand. A modern book like Counterexamples In Analysis makes these available: functions that are continuous everywhere but differentiable nowhere, a nowhere-dense set with positive measure, etc. The theorems come from somewhere, and it seems like they come from mathematicians’ intuition for the objects they’re dealing with. If the only examples that you’ve dealt with share a certain look and feel, perhaps it’s unavoidable that your mental picture will differ from what logic alone would tell you.

Second, Hawkins’s book puts Georg Cantor’s work in greater perspective, at least for me. This business about finding the conditions under which Fourier series can be integrated term-by-term is a fundamentally useful pursuit, and Cantor’s work involved constructing interesting counterexamples of bizarre sets with weird properties. Cantor’s work is often presented as fundamentally metaphysical in nature; his diagonalization argument is used to prove, e.g., Gödel’s incompleteness theorem. It’s rarely presented as part of a program to make mathematicians’ lives easier.

Perhaps Hawkins gets here (I’m only a fraction of the way into his fascinating book), but I wonder what the experience of developing these counterexamples did to later mathematical practice. Did it make future mathematicians in some sense hew more closely to the words in their definitions, under the theory that words are a surer guide to the truth than intuition? Or is that not how it works? If the definitions don’t match your intuition, perhaps you need to pick different definitions. After all, the definitions are tools for human use; you’re not plugging your Platonic bandsaw into a Platonic power outlet to help you construct a Platonic chest of drawers. If the tool doesn’t fit in the hand that’s using it, it’s not much of a tool.

I hope that’s how Lebesgue integrals end up working, as the story unfolds: the definitions function as you’d expect them to, so you can use them freely without having to preface every assertion with a pile of assumptions.

What I don’t know — what my dilettante’s understanding of integration thus far hasn’t totally answered — is whether Lebesgue integrals are really, truly, the “outermost limits” of the integral concept. I understand that the following is how modern measure theory works. We start with some set — let’s say the set of all infinite sequences of coin tosses, where a coin toss can — by definition — only result in heads or tails. Then we choose some collection of subsets of that set to which we’re allowed to attach meaningful ‘measure’ (think ‘weight’ or ‘length’ or ‘volume’ or ‘probability’). Maybe we allow ourselves to consider only finite sequences of coin tosses, for instance. Talking about the probability of an infinite sequence of coin tosses would be, under this thought experiment, literally impossible: the system would assign the words no meaning. Finally, we attach a rule for the assignment of probabilities; maybe we say that any sequence of n coin tosses has the same probability as any other sequence of n coin tosses; this “equiprobability” assumption is how we typically model fair coin tosses.

These together — the set, the collection of admissible subsets, and the measure attached to each admissible subset — constitute a measure space, or, in a particular context, a “probability triple”. (When we’re talking about probabilities rather than more general measures, the probability of the set — the probability that something happens — must equal 1.)

Now, why would we pick a collection of subsets? Why not just stipulate that we can meaningfully attach a measure to every subset of the set? It turns out that this is in general impossible, which I find fascinating; see the Vitali set for an example. I don’t know at the moment whether non-measurable subsets arise from countable sets (e.g., our infinite sequence of coin tosses, above), or whether they can only arise from uncountable sets. In any case, the upshot is that you always have to specify a set, a collection of admissible subsets, and a measure that you’ll attach to each subset.

There are several directions that you can go from here. One is to restrict your collection of subsets such that all of them are measurable; this is how you end up with Borel sets, or more generally how you end up with σ-algebras. And that’s where I’m curious: can we show that there is no more useful way to define an integral than to define a σ-algebra of subsets on the set we care about, then define the Lebesgue measure on that σ-algebra? Do σ-algebras leave out any subsets that are obviously interesting? Is there some measure more general than the Lebesgue measure, which will fit more naturally into the mathematician’s hand? Or can we prove that the Lebesgue measure is where we can stop?

In order to make statements about integrals of all kinds, we’d need to define what an integral in general is, such that the Riemann integral and the Lebesgue integral are special cases of this general notion. I gather that the very definition of “measure” is that general notion of integral. A measure is a function that takes a subset of our parent set and attaches some weight to it, such that certain intuitive ideas apply to it: a measure is non-negative (i.e., the weight of an object, by definition, cannot be less than zero); the measure of the empty set must be zero (the weight of nothing is zero); and the measure of distinct objects, taken together, must be the sum of the measures of the objects, measured separately. We call this last axiom the “additivity axiom.” You can add other axioms that a measure should intuitively satisfy, such as translation-invariance: taking an object and moving it shouldn’t change its measure.

The additivity axiom introduces some problems, because infinity is weird. Do we use the weaker axiom that the measure of the sum of two objects must be the sum of the measures of the two? Or do we use the stronger one that the measure of a countable infinity of objects, taken together, must equal the countable sum of the measures of each object? These alternatives are described, respectively, as “finite additivity” and “countable additivity”. One reason to pick finite additivity is that finiteness is, in general, easier to reason about, and has fewer bizarre gotchas. But finite additivity is also not as far-reaching as what we need. You can’t reach infinity by a progression of finite steps, so finite additivity doesn’t allow you to talk about, say, the probability that a limit of some infinite sequence is thus-and-such; without that ability, you can’t prove theorems like the strong law of large numbers. (I’m pretty sure you can prove the weak law using only finite additivity.)

So that would seem to be one answer to the question of whether Lebesgue integrals are the be-all and end-all of the idea of an integral: it depends upon how sure you want to be in your axioms. If you’re willing to introduce all the weirdness of infinity, then go ahead and use countable additivity. And it’s probably the case that there are intuitively true statements to which most everyone would agree, which can only be proved if you admit countable additivity.

The idea of a non-measurable set also rests on the Axiom of Choice. (I can’t prove it, but I imagine that — like so many things — the existence of a non-measurable set is equivalent to the Axiom of Choice.) So if you reject the Axiom of Choice — which Cohen and Gödel’s proofs allow you to do, free of charge — you could make all your sets measurable. But presumably there are good, useful reasons to keep the Axiom of Choice.

So maybe — and I don’t know this, but it sounds right, and maybe Hawkins will eventually get there — we arrive at the final fork in the road, from which there are a few equally good paths to follow through measure theory. We can toss out the Axiom of Choice and thereby allow ourselves to measure all sets; we could replace countable additivity with finite additivity and accept a weaker, but perhaps more intuitive, measure theory that doesn’t use the Axiom of Choice at all; or we could go with what we’ve got. In any case, the search for the One Final Notion Of Integration would probably be the same: keep looking for counterexamples that prove that our axioms need reworking. That will probably always mean looking for obviously true statements that any sound measure theory ought to be able to prove true, and obviously false statements that any sound measure theory ought to be able to prove false. The ultimate judge of what’s “obviously true” and “obviously false” is the mathematician’s. A similar approach would be to come up with a system of axioms from which all the statements that we accept as true today can still be derived, but from which, in addition, we can derive other, interesting theorems. Again, the definition of ‘interesting’ will rest with the mathematician; some interesting results will just be logical curiosities, whereas others will prove immediately useful in physics, probability, etc.

Phew. This has been my brain-dump about what I know of measure theory, while I work through a fascinating history of the subject. Thank you for listening.

It’s quite amazing to me

…that we’re replaying the crypto wars. Terrorism and child pornography are evergreen weapons for scare mongering, it seems. If you’d like to know when I first got politicized about technology, Google for [Clipper chip].

Mr. Cole offered the Apple team a gruesome prediction: At some future date, a child will die, and police will say they would have been able to rescue the child, or capture the killer, if only they could have looked inside a certain phone.

His statements reflected concern within the FBI that a careful criminal can shield much activity from police surveillance by minimizing use of cellphone towers and not backing up data.

The Apple representatives viewed Mr. Cole’s suggestion as inflammatory and inaccurate. Police have other ways to get information, they said, including call logs and location information from cellphone carriers. In addition, many users store copies of a phone’s data elsewhere.

During the hourlong meeting, Mr. Sewell said Apple wasn’t marketing to criminals, but to ordinary consumers who store growing amounts of data about themselves on smartphones and are increasingly suspicious of tech companies. Many of those customers are outside the U.S., the Apple representatives said, where phone users want to shield information from governments that are less respectful of individual rights.

If the government wants more information from Apple, the company representatives said, it should change the law to require all companies that handle communications to provide a means for law enforcement to access the communications.

Mr. Cole predicted that would happen, after the death of a child or similar event.

More than once, Mr. Cole suggested there had to be a technical solution—a way to design a phone so that police, with a court order, can access information, without compromising security.

“We can’t create a key that only the good guys can use,” Mr. Sewell responded.

(Cached copy. Or, at least for the next few days, you can get it by going to news.google.com and searching for ‘Apple and Others Encrypt Phones, Fueling Government Standoff’.)

Come work with me!

I work for Akamai, the company that everyone who uses the Internet uses every day of his or her life (we carry 15-30% of all web traffic), yet no one (not even geeks, in my experience) knows that they’re using. What we do is super-important, and the scale is just fantastic. In truth, there are only a few companies in the world that would allow you to work at this kind of scale. For an engineer, it’s great work.

Also, I work there, and I’m pretty great.

These two facts together should get you, if you’re a geek, to want to work at my company. We’re hiring for a role that would allow you to work with me every single day.

If I haven’t sold you yet, how about if I tell you that the company pays pretty well?

You should email me about applying to work at Akamai.

Some quick reviews of books I’ve read recently

Preface: I’m just catching up on books that everyone else read a decade and more ago. So sue me.

  • John Cheever, The Wapshot Chronicle

    I’m still sort of confused about this novel (Cheever’s first, after a career spent more-famously writing short stories). It’s several kinds of stories rolled into one: part semi-Biblical novel about one family; a tale of the Wapshot kids’ growing up and, well, boning; and maybe an exploration of male paranoia.

    It starts out feeling like it’s going to be some fusty novel about quaint rural life in some old-school sleepy New England town. (Cheever was born in Quincy.) There are little hints early on that it won’t be so, like when Cheever mentions in an aside that the kids are occasionally going out whoring. And then there’s the bizarre grandmother, who holds the rest of her family under her sway through the threat of withdrawing her inheritance; this inheritance depends upon her grandchildren producing male offspring.

    Then that matriarchal Sword of Damocles, so far as I can tell, disappears from the rest of the book; the matriarch herself does too, mostly. The kids go off into the world to make their fortune and escape from their little town; one goes to New York and another to D.C. One of them marries a beautiful woman who is, forebodingly, bound tightly to her mother, who also dangles her family via some invisible string. The beautiful woman, not to put too fine a point on it, goes crazy at some point. Meanwhile, the other brother marries another woman who goes crazy in her own way, writing him a Dear John letter from back home, whence she’s returned.

    But no matter: by the time the novel is done, both couples have gotten back together, for reasons that are completely unclear. One of the joyful reunions involves a completely unbelievable deux ex machina.

    This is all so oddly plotted that I have to imagine it was deliberate — a book like this couldn’t turn out the way it does by accident — but the allure was completely lost on me. There’s something in here about being a man, surely, and about male feelings of powerlessness; that’s not to be scoffed out, despite a feeling (voiced by a long-lost friend many years ago) that books about male emotion ought to take a backseat for the next few decades so that books about women can take the spotlight. (We were discussing Philip Roth at the time — The Dying Animal specifically, if memory serves.)

    Books about men are important, and can generally be worth reading. I’m not so sure about this one, though.

  • Amy Poehler, Yes Please

    This book made me laugh uncontrollably on a few occasions on a cross-country flight recently, to the point that I was feeling spasms in my chest as I tried to avoid annoying my seatmate. It also made me cry repeatedly: Poehler seems to genuinely love life and love her family, and her love is contagious (at least if you’re a sentimental fellow like me).

    This is a book about Poehler’s rise through the Chicago comedy scene, through to Saturday Night Live and Parks and Recreation (a television show that I recommend in the highest terms, at least from season 2 until Rob Lowe and Rashida Jones left). It’s a pure delight. Imagine your most ebullient friend gushing about the amazing life she’s led in little ten-minute essay chunks, and you’ve got a good sense of Poehler’s book. I started it right around the start of a 5.5-hour flight, and finished it maybe two hours from the end. You should buy it, read it, and love it.

  • Jeffrey Eugenides, Middlesex

    This novel is partly a historical epic, spanning several generations of one Greek family from its hasty departure out of Smyrna as that city burned to cinders in the 20s, through its arrival in Detroit as that city did the same in the 60s. But it’s also partly an emotional study of our intersex narrator. And in the process of studying him, it’s a scientific walk through intersex issues generally.

    Honestly, I’ve never read anything like it. No book I’ve ever read has been both grandly historical and richly character-driven. It’s got the depth of character of a Love in the Time of Cholera, with the (never-dry) historical arc of a work of nonfiction. It’s breathtaking.

  • Jeffrey Eugenides, The Virgin Suicides

    This is completely unlike Eugenides’ later novel. I think it’s fair to call this a very, very black comedy, though it’s so black that sometimes I don’t know whether it’s a comedy. We know early on in the novel that all (sic) of the daughters in this one family will, by the end, have committed suicide. We watch the story of their gradual deaths through the retrospective eyes of a man who grew up in their neighborhood and — like all his street-mates — lusted after them mightily, if confusingly. They’re identical-looking beautiful blonde girls, or at least they’re identical when viewed from afar. And as it turns out, afar is the distance from which awkward teenage boys will view them. So we’re listening to a man in his 40s or 50s describe his memories of girls whom he was mostly too scared to touch 30 years earlier.

    And they all die. But I’m inclined to say that the fact of their deaths is not even the point of this book. I mean, yes, it’s the central frame off which the rest of the story hangs. The girls’ parents slowly withdraw into their home, progressively allowing it to rot while the neighbors all watch in stupefied horror. No one really does anything about their decline, which is maybe the point — or maybe not. I really think the point of this book is in the endless little details that add up to something that is just alarmingly funny. For some reason, for instance, a couple parts of these sentences caught me:

    The Pitzenbergers toiled with ten people — two parents, seven teenagers, and the two-year-old Catholic mistake following with a toy rake. Mrs. Amberson, fat, used a leaf blower.

    First, “two-year-old Catholic mistake.” And there’s something just beautifully economical and perfect in “Mrs. Amberson, fat”. Given the chance to express that idea, 99% of humans — myself included — wouldn’t have given it a second thought: that would have been “Mrs. Amberson, who was fat” or “the fat Mrs. Amberson” or a dozen similar alternatives. But no, she’s “Mrs. Amberson, fat”. That’s an immeasurably better sentence.

    I can’t quite articulate whether that’s why The Virgin Suicides is so dry, and so funny. But it is both. And the accumulated effect of page after page of this dryness is that you’re laughing uncontrollably while an entire family is dying. This left me permanently off-balance throughout. Other examples start you in one place and end abruptly in a way that makes you back up and ask, “Wait, really?” E.g.,

    Our interview with Mrs. Lisbon was brief. She met us at the bus station in the small town she now lives in, because the station was the only place that served coffee.

    I can’t get over the hilarity of a town so rotten that the best coffee is to be found in a bus station. This immediately calls to mind at least four forms of grey, bleak disgust: the sort of town about which this would be true, the sort of bus station that this sort of town would have, the sort of coffee that they’d serve there, and the sort of person who would rather meet you at a bus station for coffee than pour you a cup in her own home. And it’s just two sentences. I can only imagine that Mr. Eugenides pared and pared and pared and pared some more, until the bare minimum number of words were left to convey the laughably dismal world he wanted. And then he moved on to paint another scene — as briefly as possible, but no more briefly.

    Even the sex scenes are out of some parallel-universe science-fiction/fantasy dystopia:

    He felt himself grasped by his long lapels, pulled forward and pushed back, as a creature with a hundred mouths started sucking the marrow from his bones. She said nothing as she came on like a starved animal, and he wouldn’t have known who it was if it hadn’t been for the taste of her watermelon gum, which after the first few torrid kisses he found himself chewing. … It was as though he had never touched a girl before; he felt fur and an oily substance like otter insulation.

    Everything about this passage is off-balance. He ends up chewing her gum? Otter insulation? This isn’t a sex scene, and it’s not the least bit sexy. Maybe it initially promises to be, in that you start out thinking that this she-beast is a Hall and Oates-style man-eater. But then you get to otter insulation. There is nothing sexy about otter insulation. Also: “insulation”? Any word that legitimately fit there — in an ordinary world — would have been minimally sexy. Consider ‘pelt’ or ‘fur’ or even ‘quills’. ‘Insulation’, by contrast, is the least sexy word that bears any relation to physical reality there. It turns this young woman’s body into construction equipment.

    Everything, just everything about this novel is intended to leave you a few degrees off plumb. Like Eugenides’ other novel, above, I’ve never read anything like it, but what’s remarkable is that The Virgin Suicides and Middlesex, both masterpieces, have so little in common. It would be churlish to demand a similarly masterful, similarly sui generis third act; if Middlesex and The Virgin Suicides are all Eugenides ever gives us, we should count ourselves blessed.

  • Joan Didion, Democracy: A Novel

    If you’ve read Didion’s nonfiction political works from the 80s, this is exactly the novel you should expect. And, to be clear, you really really need to read Didion’s nonfiction political works from the 80s. Particularly After Henry. She’s just icily cynical about the world.

    Imagine a menacing episode of a soap opera, where the characters say virtually nothing to each other because nothing is left to say, and where virtually all of the soap opera’s menace comes from chilly atmospherics. That’s Democracy in a nutshell. There’s a U.S. senator and his wife; there’s a military attaché who spends most of his time in the air making deals about unspeakably deadly military hardware. There’s the politician’s daughter, overdosing in a miserable flophouse. And all throughout, as backdrop, there’s the evacuation of Saigon, spreading nameless fear over everyone.

    It turns out that Didion is just as keen an observer of fictional political characters as she is of real-life ones.

  • Steve Martin, Born Standing Up

    I got two big messages from Born Standing Up. First, Steve Martin worked very, very hard, for 18 years, to go from nothing (well, to go from working at a Knott’s Berry Farm) to the level he eventually attained, where there are few people more widely beloved. I’ve come to think over the years that working very, very hard is the only answer to most questions of how to be successful. Reading Martin’s autobiography — since that’s what this is; or at least it’s the first volume of one — made me feel incredibly lazy.

    The second thing it taught me is that you can take major risks like he did if you have no one depending on you. I wouldn’t be surprised if the bulk of the world’s risky accomplishments came from single people, like Martin, in their twenties. I’d love to ask the man whether he thinks he could have achieved what he did had he been married with children.

    The book only reinforces my love of his standup work from the 70s, which is the era this book focuses on; it ends with his writing for the Smothers Brothers and starting on the production of The Jerk. (The book, by the way, confirms my suspicion that his stand-up bit about how he was “born a poor black child,” and how he really found himself when he heard his first Mantovani record, pre-dates the movie based around that premise.) It’s the record of a man starting out not knowing what he’s doing, and slowly accumulating fame by playing the game and refining his act endlessly. Perform in several thousand nightclubs and you maybe — again, with a lot of hard work — will eventually meet the guy who will open the door that eventually opens another door for you. But of course Martin wasn’t just doing the same thing over and over; he documents tinkering with his jokes night after night, adapting them to the subtlest changes in his audience’s mood. Then he’d return to his lonely hotel room and agonize over how to make his act better until sleep overtook him.

    This and the Poehler book make me feel both inspired and depressed. They feel like people who’ve worked exceptionally hard, taken exceptional chances, and ended up doing exactly what they love every day of their lives. Perhaps they’ve downplayed (deliberately or otherwise) the drudgery involved in doing any job, even the dream jobs they ended up in; or perhaps not. In either case, I’d like to take the Martin and Poehler books and use them to build a life that’s worth living.

What will be the next gay-rights-style debate?

A thought: circa a century ago, say, the thought that gay people deserved equal treatment was probably so absurd that it didn’t even need to be discussed, because no one thought it was true. Then at some point it became a debatable proposition, and now it’s so obviously true that it doesn’t need debate; the generation that believes gay people don’t deserve equal rights will die off soon enough. There are plenty of other obvious examples of this phenomenon. Consider intermarriage between the races, for instance. If you go back far enough, probably religious toleration would fit in the same bucket.

So I wonder what sort of issues are, at this very moment, in the first phase of that evolution: things that even we (who consider ourselves more enlightened than our benighted ancestors) consider so absurd or so obvious that no one even bothers to discuss them, which will eventually become topics of vigorous debate, and will later on become obviously true or obviously false, respectively. I can dream that maybe “the nation-state is a sensible grouping of human beings” or the related “it is right and just that we treat those who live on the other sides of an arbitrary border differently than we treat our own families” will one day become debatable. Or maybe even those are too explicit; maybe the sort of propositions that we take for granted and will one day reject are exactly those propositions that I couldn’t even think to write down.

Phrased this way, it could be seen as a hopeful question — part of our habit of viewing history as a vanguard marching ever forward in the direction of social liberalism. (I think this might be “The Whig Interpretation of History”, after the book of the same name by Herbert Butterfield. Though it sounds slightly different.) I can see this going in a very illiberal direction as well, however. E.g., maybe there was a time when it was considered obvious beyond the point of discussion (or even of conscious thought) that we lived in something called a “society” in which we were more than just disjoint libertarian billiard balls colliding inelastically into one another. Maybe it was once considered obvious beyond the point of discussion that the main way in which an industrial democracy cares for its least fortunate was by way of its government, which was largely expected to spend money wisely. And so forth. You can imagine any of these slowly becoming less true. And when they become less true, you can imagine them becoming less true in an exponentially-increasing way: if all your friends believe that the less-fortunate are largely shirkers living off the dole, that might cause you to feel the same way (or maybe it’s not causal).

So there’s nothing necessarily liberal about this sort of change, if indeed it happens. I’d like to talk to an actual historian about how one might measure this sort of change. I imagine it would be difficult: almost by definition, the ideas which are believed so widely that they’re beyond the reach of conscious thought are those which few people will ever bother to write about. Extracting earlier societies’ unconscious beliefs might involve digging into the unspoken assumptions behind what they are saying.

Reminding myself how beautiful statistics is

As I think I’ve mentioned here before, my partner is taking a biostatistics course and thereby reminding me of how much I loved this stuff. And I’m reminded of the Galton quote about the Central Limit Theorem:

I know of scarcely anything so apt to impress the imagination as the wonderful form of cosmic order expressed by the “Law of Frequency of Error.” The law would have been personified by the Greeks and deified, if they had known of it. It reigns with serenity and in complete self-effacement amidst the wildest confusion. The huger the mob, and the greater the apparent anarchy, the more perfect is its sway. It is the supreme law of Unreason. Whenever a large sample of chaotic elements are taken in hand and marshalled in order of their magnitude, an unsuspected and most beautiful form of regularity proves to have been latent all along.

It’s not only beautiful, but it’s obviously extremely useful. Yet, given how often I’ve failed to explain how a random sample of a couple thousand people can adequately capture the political views of a nation of 318 million, clearly there’s something mysterious and objectionable about it. For that matter, given how many people took umbrage at Nate Silver’s election forecasts, even though basically all he did was average poll data, it seems like this antipathy to statistics is pretty widespread; statistical laws explain exactly where, and under what conditions, you’d expect individual chaos to yield collective order, yet people really seem to recoil from the thought that their collective actions might be rule-governed.

It really does often feel like I’m possession of a kind of occult knowledge that everyone could learn but few choose to. And I’m nowhere near the level of statistical knowledge that I want to attain. Even just the bit of probability and statistics that I know is enough to resolve a lot of mental muddle.

One of the clearest things about measure theory I’ve ever read

Check out Terry Tao’s measure-theory book, starting with ‘let us try to formalise some of the intuition for measure discussed earlier’ on page 18, through to ‘it turns out that the Jordan concept of measurability is not quite adequate, and must be extended to the more general notion of Lebesgue measurability, with the corresponding notion of Lebesgue measure that extends Jordan measure’ on p. 18.

I’ve understood for some time that there’s a notion of “non-measurable set”, and that you want your definition of ‘measure’ to preserve certain intuitive ideas — e.g., that taking an object and moving it a few feet doesn’t change its measure. I didn’t understand that there was any connection between non-measurability and the axiom of choice. Tao’s words here are some of the first that have properly oriented me toward the problem that we’re trying to solve, and the origins of that problem to begin with.

My partner is taking a biostatistics course, which is reminding me of how much I loved this stuff at CMU. I’m inclined to find a course in measure theory around here. We have a university or two.