Polimath

My dad and I went to the recent Brown/Whitman California gubernatorial debate here at UC Davis. It was fun seeing “democracy” live and up close. One of the candidates twice repeated an old saw:

One definition of insanity is doing the same thing over and over and expecting different results.

The quote was misattributed to Einstein, and it’s often misattributed to Ben Franklin, Mark Twain, and others. Various folks take issue with this pop-culture “definition” in different ways.  As a recovering mathematician, I’ve repeated it myself, and it used to make some sense to me.

However, as I’ve tried to move from brittle assumptions (knife-edge mixed equilibria, trembling hand-perfection) in my models to more robust assumptions (quantal response equilibria, non-parametric tests) this truism has rung less true.

Suppose you hit a subject with a treatment.

A physical scientist tends to have objects of study that have simple internal states.  The aggregate systems are complex, but the smallest observable parts of the system are very simple.  The simplicity of the set of states means that I have a good chance of being able to choose a sample from those states that is sufficiently random so that there is little or no correlation between states and my variables of interest.  No correlation means that we get unbiased estimates of the parameters for models connecting my variables of interest.  I might run a physical experiment, say, n times to assure that we have a good sample and to compensate for measurement uncertainty.  I don’t really expect running it another n times to show me anything new; that would be crazy.

A social scientist has objects of study that are vastly more complex. A person has far more possible internal states than an atom, a mole of gas, or a cell.  More germane, we can’t easily be sure of sampling from those internal states randomly, so we constantly face the threat of internal states being correlated with our variables of interest, which leads to bias if not statistically corrected.

Can we just set n higher?  No. If we had a better understanding of the internal states of a person, we might be better able to sample randomly from those states.  At least for now, however, we do not grok the data-generating processes within people as well as we do those in chemistry.  Samples would have to astronomically large to overcome the selection bias in our samples of the internal states of a person.

I’ve heard this “definition” many, many times.  Each time before I kept my mouth shut.  This time I had to speak out.  Keeping silent would have driven me crazy.