Patil takes up the necessary and generally thankless task of writing a “big-think piece”.  It’s necessary because, with all the recent talk about Data Scientists, it would be easy for some to see “Data Science” as a recent entry in the long history of business fads.  Business fads tend to indulge in one of two sins: oversimplifying, or being so general that anything counts.  Both are sins to which Data Science advocates are susceptible.  An advocate of Data Science might oversimplify by giving a recipe or IT shopping list for doing Data Science; Patil avoids this triteness by emphasizing the wide diversity of the good teams he’s built or worked with.  Alternatively, one could sin by making anything count as Data Science.  Commenter “Verbose” anticipates this problem with his erudite critique of BDST: “Same technical data analysis, new bullshit name“.  Patil avoids this and provides a blueprint for others to do the same.

When discussing the etymology of “Data Scientist” Patil writes that “Research Scientist” would not be as appropriate a term for this profession:

“Research scientist” was a reasonable job title used by companies like Sun, HP, Xerox, Yahoo, and IBM. However, we felt that most research scientists worked on projects that were futuristic and abstract, and the work was done in labs that were isolated from the product development teams. It might take years for lab research to affect key products, if it ever did. Instead, the focus of our teams was to work on data applications that would have an immediate and massive impact on the business. (emphasis added) The term that seemed to fit best was data scientist: those who use both data and science to create something new.

Elsewhere Patil provides a solid definition of Data Scientist, but this paragraph encapsulates the concept just as well:  A Data Scientist uses data and science to have an immediate and massive impact on the business.  Just moving data around?  Not data science.  Have an impact in the vague future?  Not data science.  Improving entirely on the margins?  Not (all of) data science.  Holding the Data Scientist’s feet to the fire — asking “How does this immediately and massively impact our business?” — provides accountability and hence focus for the team.

Writing a “big-think piece” is also a thankless task: the breadth of the topic means that it’s easy for critics to find something to criticize as being presented too simply.  This ignores the contribution of providing a view of the new discipline from space, showing all of it as a piece, and showing (if briefly) how the disparate parts fit together.  I was glad for a look at a map for this road I’m traveling.

Overall BDST is short, shorter than I would have liked.  (I’m glad that Patil is sharing more of his experience through other venues.)  The advice Patil gives about Data Science, forging teams, and hiring Data Scientists seems both specific and useful; I’ll post again as I have occasion to use his advice.

Recommendation: “Building Data Science Teams” is short, but with enough good ideas as to be required for anyone in business intelligence, internal data analysis, or applied computational modeling and prediction.

In theory, a panel takes place in a room where 3-300 (median = 10) people watch three to five papers get presented by their authors.  Then a discussant, who reads the papers in advance, comments on the papers both to draw connections among them and to stimulate conversation among the attendees.  Then the audience asks questions and offers feedback to the authors.  The whole panel takes about 1 3/4 hours.

Although panels comprise most of the scheduled events at a conference, they are not the best reason for scientists to attend conferences and they are far from the most rewarding part of a conference.  Panels are often poorly attended.  The papers in a panel often have very little to do with each other.  The discussant may not receive the papers until days or moments before the panel, if at all, and even so the comments may focus more on typography than on big ideas.

Panels are a party game. They are an excuse to get smart people, who are interested in similar things, together in a room talking. Put a bunch of clever folks together and strange, wonderful, unpredictable things happen.  A conference is mass planned serendipity.

The largest conference benefits to my research have happened when I have not been at panels: between panels, skipping panels, into the evening and the night. It’s the networking, but not “networking” in the Machiavelian, sales-person sense.  It’s the comment on my paper that someone was a little too shy to offer in front of everyone, the comment that helps me recast the paper so it will place higher.  It’s running into the same person at three panels and finally discovering we would love to work together on some research.  It’s the dinner outing that leads to an invited talk or an interview.  It’s the shared coffee followed up with a Facebook friending that leads to a new real friendship.

All of this is made possible by panels, but it’s not the direct result of the panels.  So when someone tells me they didn’t go to a lot of panels, I understand that they probably got a lot of professional good out of the conference.

Also, I had a lot of fun at the Space Needle.

It is regular sport for Bayesians to criticize frequentist confidence intervals as unintuitive, usually misinterpreted, and based on what are usually unjustifiable assumptions.  There are good reasons for this:  they are unintuitive, usually misinterpreted, and based on what are usually unjustifiable assumptions.

Confidence intervals take into account the uncertainty we have in trying to describe a population given that we only observe a random sample.  Maybe our sample is representative of the population, maybe not.  The smaller the sample, the more likely it is that, through random variation, we got a sample that suggests a relationship (causes us to reject the null hypothesis) when really there is no relationship (the null is true.)  If we take 100 random samples of the same size and for each construct a (different) confidence interval, how many will contain the true value of the parameter?  We expect that 95 of those constructed intervals will contain the true value.

We don’t have 100 samples of the same size, so this is not the question we want to answer.  Here’s what we want to know:  What is the probability that the true value is in the interval?

There are a variety of very good, sound reasons why the frequentist approach does not make sense. I understand the difference between believing that the true parameter is “fixed and known to God” (per the frequentist assumption) and a random variable (the Bayesian assumption.)  I agree completely that a confidence interval answers the wrong question.

It doesn’t matter.

For large samples and given the regularity conditions of maximum likelihood estimators, the marginal posterior distribution for a single parameter is approximately normal with a mean equal to the MLE and standard deviation equal to the standard error.  Under these conditions, maximum likelihood and Bayesian estimators give you the same inferences.

Bayesian researcher perspective: Suppose I want a credible interval for a parameter where I have a lot of data and the model is well-behaved but I don’t have convenient code for drawing posteriors.  I could estimate the MLE and use the resulting confidence interval as an approximate credible interval. My friend and colleague Jeff Gill calls this the “lazy Bayesian” approach.  Lazy, efficient, tomato, tomahto.

Bayesian consumer perspective: Suppose you are reading an article where there is plenty of data and a well-behaved model, but the author provides frequentist confidence intervals.  You can treat them as approximate credible intervals.  Easy peasy.

Pragmatist perspective: As long as the conditions are met, you can go on “misinterpreting” confidence intervals.

This “lazy Bayesian” approach is not limited to making simple inferences about single parameters.  King, Tomz, and Wittenberg explain how to generate draws from the approximate posterior (as opposed to approximate draws from the actual posterior via MCMC) and make any inference a Bayesian can using the output from maximum likelihood estimation.

As a Bayesian, I advocate strongly for increasing the use of Bayesian methods.  However, we should be careful to avoid overselling the advantages.

Bayesian estimation addresses the “small problem”

This is right and wrong.

Maximum likelihood estimators (MLEs) leverage large in two ways.

1. Making inferences about parameters.
2. Checking model fit.

The goal of #1 is to answer questions like “Is ?” or “What is the shortest interval that has a 95% change of containing ?”  MLEs let us take a stab at answering* this. A large number of observations means we can apply the Central Limit Theorem, which means we can use the standard errors to test simple hypotheses and build confidence intervals.  The goal of #2 is to justify our choice of model after the fact.  Presumably we had a good, substantively-informed theory to justify our choice of model, but it’s nice to be able to say afterward “See? The models fits well, so we weren’t crazy to choose this model.”

How many is a “small” number of observations?  It’s difficult to say.  Long (1997) gives some guidance about this, suggesting that 100 is a minimal number for an MLE, and that one needs at least 10 observations per parameter in the model.  However, we never really know if this is enough to be able to trust our inferences and fit.

What happens if we have too few observations?  Both #1 and #2 become unreliable.  We have too little data to assume that the Central Limit Theorem has “kicked in,” so our point estimates have more uncertainty, which means our standard errors are too small**, and therefore any inferences are unreliable.  Our tests for model fit are similarly starved for information, so any post-hoc model justification will be difficult.

What does Bayes fix?  Bayes estimators are finite-data estimators:  more data gives more accurate estimates, but the measures of uncertainty of those estimates are reliable regardless of the amount of data.  This means that inferences are reliable regardless of the size of the data set.  Want to know if but our $\latex n$ is small?  No problem for a Bayesian.

What about model fit?  Bayesians have the same tools as frequentists for checking model fit and they also have the numerical and graphical analysis of posterior predictive distributions.  As with inferences about parameters, more data is more information and thus is preferable.  However, any tests of model fit are still reliable.  As a purely practical matter, if we have a very small data set we probably will not be able to conclude anything about model fit from the data.  This puts the burden back on the substantive/theoretical argument for the model form.  If we understand the data-generating process very well, great; if not, then this part of our argument will need extra scrutiny.

Bottom line: Bayesian estimators don’t create more information.  However, they do let us correctly identify how sure we are about the inferences we draw.  That’s a clear improvement.  Other Bayesians aren’t helping by overselling it.

===

* Don’t get saucy with me about how frequentist confidence intervals (CIs) either contain or don’t contain , implying that it’s meaningless to talk about the probability of being in the interval. CIs are better than commonly described — details in my next post.

** In theory our standard errors could be too small, too large, or correct.  We can appropriately account for this additional uncertainty by increasing the size of our standard errors (think “put a confidence interval around our confidence interval”) except we can’t know how much to increase them.  See “reasoning, circular”.

The way to be sure that a statement about politics is true is to make a formal argument: explicitly state the assumptions and use rigorous logic to create a 100% valid argument from assumptions to conclusions.  Euclidian geometry was a popular touchstone of Truth^TM for two thousand years…until Gauss proved that one of Euclid’s assumptions was arbitrary.

Oddly enough, most of the mathematicians I know do not assert that math reveals Truth^TM.  For a set of statements to be true, they must not disagree.  Gödel’s showed us that we cannot be sure that math does not contradict itself.  Accepting this means abandoning the claim that 100% truth can be generated analytically or using any logical system. For many mathematicians (not including probabilists or statisticians) once something is uncertain, all bets are off.

Analytic formal modelers know this, but assert that deduction (assumptions + logic => conclusion)  is better than statistical induction (theory + data => conclusion).  Let’s take a look at this analytically, shall we?  Consider this statement that is true simply as an application of the definition of conditional probability:

Analytic modelers use a deductive chain of reasoning, which has a (conditional) probability of being true that approaches 1, so the probability of the conclusion being true is essentially the same as the the probability that the assumptions are true.  One critique of this approach is that all assumptions, if pushed far enough, are false.  However, that attitude underestimates the usefulness of analytic results.  If the assumptions are close to being true and the chain of reasoning is “well-behaved” (robust to small errors in inputs) then the conclusions will be close to true.

Computational (including agent-based) modelers start with clear, explicit assumptions but use an inductive chain of reasoning which is therefore less likely to be conditionally true.  However, the original assumptions are often much more realistic:

If the assumptions made in a computational model are more likely to be true (or nearly true) and the chain of reasoning is strong, then the conclusions are more likely to be true than those “proven” using an analytic formal model.

Of course, that’s a tall order to fill.  Your mileage may vary.  Whether analytic or computational approaches is more appropriate depends on the specific assumptions being made and the strength of the induction used in the computational approach.  This is a trade off that should be made mindfully, not dogmatically.

Three nights ago I had a dream in which I was being assaulted. One of the two offenders in my dream was someone from my division on the ship, someone I have not seen in over fifteen years. I woke up wondering what that was all about, but I gave it little thought as I went on through my day, my happy, productive day. Everyone has weird bad dreams sometimes. Civilians just have ones with less gray paint.

When I was on the ship I lived in, essentially, a locker room. My 38 roommates and I slept in 13 triple-stacked bunks in a berthing space the size of my current living and family rooms. Before the Navy I hated locker rooms. I wasn’t frail, but I was a geek in high school and no athlete. I hated the inevitable comparisons. In the Navy I hated showering next to a 198-lb functionally retarded man who could bench press 500 lbs and who, because he had joined many years before and outranked me, treated me like a child. I hated the regular joking about sailors being gay, complete with accompanying grabbing of each others’ bodies. Yesterday when changing at the gym I was completely comfortable in the locker room, thinking about the class I was going to teach a little later.

Here’s something weird: I never heard about any crime on the ship more serious than a regular, playground-style fight. I never heard about a sexual assault. I never saw any sign of any actual homosexuals. (My gaydar wasn’t completely oblivious, as I had been around the “family” a lot in my previous college years.) I never saw anyone shoot or get shot in the line of duty or otherwise. I never really faced an enemy. I never had to deal with a fellow sailor who was obviously a “bad guy”, someone who would, in the boring movie version of my life, be seen as any kind of villain.

There are a lot of people I know now who I like, but don’t trust. There were some people I served with whom I really didn’t like, but I trust them, and would still, with my life. And afterward it wouldn’t change how I feel about them.

Three guys I played poker with died in a plane crash. When you deploy 10,000 sailors and soldiers on metal ships with guns, bombs, torpedoes, fuel, and high voltage equipment on indifferent seas for an indefinite length of time, some don’t come back. It doesn’t matter whether it’s “peacetime” or not. They were on a E-2C Hawkeye reconnaissance plane coming in for a landing on the carrier when they got waived off. They started to gain altitude for another pass, but stalled out right after clearing the flight deck. A carrier has four screws. We never found a piece of the plane larger than a surfboard. I don’t remember their names.

My ship was in the Adriatic Sea enforcing the no-fly zone over the former Republic of Yugoslavia. We left to go home, relieved by another AEGIS cruiser who was, like us, fully provisioned and armed with Tomahawk missiles. We didn’t even make it home before we found out that the relieving ship got to fire some of her missiles. Lucky bastards! We had trained and trained — we were ready! — but we never got to fire a shot. Of course, nobody wants to kill people. Even then I knew I was lucky not to have to live with the burden of having killed.

What I didn’t realize is that, in training, in making myself mentally and emotionally ready to pull the trigger, I had already lost something. When we hurt others, we are diminished. Soldiers who are willing to do this, are willing to diminish themselves, do diminish themselves. The sacrifice is admirable, but it is a sacrifice just to prepare to kill. It does change you to prepare for war. Some changes can be more obvious, some are less so.

I am fine. I am healthy, physically and mentally. But I did not serve in combat. Many, many men and women are serving in combat now or have served in combat before. My experience was approximately 1.4 million times easier than their experience. My experience was also at least a thousand times easier than that of my wife and every other military spouse. Preparing for war takes your mind off the dangers involved; if it didn’t few people could follow through. Staying at home waiting for a loved one to return provides no such distraction. Thank you, my precious wife Dawn, for your sacrifice.

These days, I am living the dream, living the good life, the sweet retreat. I think of my father who helped identify targets for bombers during Vietnam, who now is a retired minister. I look at my children, and I hope they will serve others. If called, and am confident they will respond and serve with honor. However, I dearly hope they will never have to carry a weapon. I wish that nobody did, but some people must.

The price of freedom is eternal vigilance.

– Thomas Jefferson

One definition of insanity is doing the same thing over and over and expecting different results.

The quote was misattributed to Einstein, and it’s often misattributed to Ben Franklin, Mark Twain, and others. Various folks take issue with this pop-culture “definition” in different ways.  As a recovering mathematician, I’ve repeated it myself, and it used to make some sense to me.

However, as I’ve tried to move from brittle assumptions (knife-edge mixed equilibria, trembling hand-perfection) in my models to more robust assumptions (quantal response equilibria, non-parametric tests) this truism has rung less true.

Suppose you hit a subject with a treatment.

A physical scientist tends to have objects of study that have simple internal states.  The aggregate systems are complex, but the smallest observable parts of the system are very simple.  The simplicity of the set of states means that I have a good chance of being able to choose a sample from those states that is sufficiently random so that there is little or no correlation between states and my variables of interest.  No correlation means that we get unbiased estimates of the parameters for models connecting my variables of interest.  I might run a physical experiment, say, n times to assure that we have a good sample and to compensate for measurement uncertainty.  I don’t really expect running it another n times to show me anything new; that would be crazy.

A social scientist has objects of study that are vastly more complex. A person has far more possible internal states than an atom, a mole of gas, or a cell.  More germane, we can’t easily be sure of sampling from those internal states randomly, so we constantly face the threat of internal states being correlated with our variables of interest, which leads to bias if not statistically corrected.

Can we just set n higher?  No. If we had a better understanding of the internal states of a person, we might be better able to sample randomly from those states.  At least for now, however, we do not grok the data-generating processes within people as well as we do those in chemistry.  Samples would have to astronomically large to overcome the selection bias in our samples of the internal states of a person.

I’ve heard this “definition” many, many times.  Each time before I kept my mouth shut.  This time I had to speak out.  Keeping silent would have driven me crazy.

If you categorize political ideology into two camps, you are implicitly using a one-dimensional model, a model that puts everyone on a single line from left to right.

If you think there are two basic issues — how much the government should influence the economy and how much the government should influence social issues — then you are using a two-dimensional model, a model that locates each person on a plane.

So here’s a question:  How many dimensions are there in political ideology? This question is a lot like the question “How long is the coast of Britain?” (alt): the answer depends on how you’re going to use the answer.

If you want to explain variance in the US House and Senate, one dimension is plenty in most cases, and two dimensions is sufficient the rest of the time. If you are looking at an entire (two-year) Congress or a longer period, one dimension will do a good job most of the time

If you are trying to explain a single bill — or each of a set of single bills — the story is more complicated. On a given bill, some legislators vote for ideological reasons, some for procedural reasons, some because of interest groups or constituent pressure, some because of bribes, some because they are taking cues from other legislators, and some just make mistakes. When you are interested in distinguishing among legislators based on aggregate voting, few dimensions are needed. When you want to explain the path a bill takes, more dimensions may be required.

This has implications in three kinds of research we conduct based on the analysis of roll call votes:

1. Theorizing about ideology:  When building theories about ideology, we cannot blindly assume that ideology is one-dimensional.  Arguably there are one-dimensional policy areas, legislators, and specific bills, and the one-dimensional case is worth considering, but it is not anywhere near universal.
2. Ideology as an explained variable in empirical work:  How do parties, interest groups, constituent pressure, and other factors affect votes? When exploring factors that affect ideology — using statistical models where ideal points on the “left-hand side” — we should not ignore that there may be multiple dimensions to explain.
3. Ideology as an explanatory variable: How does ideology affect interactions with the president, the courts, and the bureaucracy? Even scholars of institutions other than Congress should remember that when ideology is on the right-hand side of the statistical model we might want to include more than one dimension.

To indulge in the snowclone, dimensionality matters when considering ideology. Make a mindful choice when modeling it.

One particularly thoughtful commenter on Wolfers’ post notes that economics combines the controversy of addressing everyday issues with the general inaccessibility of chemistry.  This conflict may make some people resist the conclusions of economists, ie. strong prior + incomprehensible evidence = small amount of updating.

The comment continues:

However, the inaccessibility of economics does not merely arise through inadvertence. As many jokes attest, economists are not merely unsentimental, they are ANTI-sentimental. An economist will often revel in the opportunity to rub people’s noses in the conclusion that their pre-conceptions are fluffy-headed poppycock. To many people (including some economists, I fear), economics appears to be less a social science than a religion, revealing to a chosen few the mighty counter-intuitive truths by which to pass judgment on a sinful world.

If I’m an accomplished scholar in another discipline, to what extent am I open and receptive to this kind of intellectual upbraiding?

Another commenter notes

My guess is at least part of this effect is reciprocity. Economists are famously bad when it comes to citing relevant research from the other social sciences. And I am not even talking about humanistic research: experiments from social psychology and statistical work from sociology are often ignored when economists do related (or even nearly identical) work. There is a widely held perception in the other social sciences that economists are, at best, disinterested in having interdisciplinary conversations–or, at worst, regularly tolerate cross-disciplinary plagiarism. My guess is that a culture of arrogance is the ultimate cause.

These are legitimate concerns, and there are reasonable rejoinders, but I think there is a more compelling explanation than “economists are a$$holes.”

When we develop a theory, we throw away some of the details about the world in order to make the theory simple enough to understand.  This abstraction is an art: there are no hard rules and too few guidelines on how to make useful theories.

Suppose that there is only one choice: how much detail to discard.  One can (grossly over-)generalize by saying that economists throw away more detail, ignoring things like irrational behavior, while other social scientists throw away less detail.  By simplifying more, the economists gain the ability to apply very technical tools to generate results that are very reliable given the assumptions they made.  By simplifying less, other social scientists are left with questions that more closely resemble reality but are harder to analyze.

Internal validity versus external validity.  Tractability versus verisimilitude.  We’ve heard this song and dance before.  The implied question is something like Heisenberg’s Uncertainty Principle:

Hypothesis:  (time to analyze) * (problems with applicability) >= (some constant)

Does this hold?  Actually, I am optimistic that it doesn’t.  Regardless, we all face choices when trying tell a story that explains something.

I don’t know if I’ll make the right decisions. I cannot know, even afterward, whether I did.  It turns out that it would take a very long time to check.  Constantinos Daskalakis has shown that it would take a computer as complex as the entire universe longer than the lifetime of the universe to solve for Nash equilibria of many common games, like how to invest for retirement. There all I am trying to do is optimize money.  When I read to my son I am doing something harder:  I am trying to influence his education, his confidence, his happiness, and my relationship with him, all without being so bored that I fall asleep.  Most of the decisions we make all day are too hard to solve.

Of course, I can simplify each of these decisions to make them tractable. Any simplifying assumptions I make are wrong, and therefore I am answering the wrong question.  I can make specific, certainly wrong assumptions, I can approximate a bunch of decision characteristics by combining them into one or more stochastic elements in a simpler game.  Either way, I am limiting my rationality, or really acknowledging that my rationality is bounded. When I make one of my decisions I am not answering the question before me but rather a different, simpler, but wrong question.

You might object.  “Once we give up rationality, we give up prediction.  There are no limits to how we can be irrational.”  How does the joke go?  “The difference between genius and stupidity is that genius has its limits.”  There is no limit on the answers we can get if we allow our models of human behavior to admit irrational behavior.  If you allow for heuristics, anything could happen, so we know nothing!

True, anything could happen, but we can still note that some things happen more than other things.  Data can guide us.  We can examine which ways of simplifying are likely and which are unlikely.  Assigning likelihood statements to different behaviors is the statistician’s strategy.  Some steps along this route are taken by quantal response theory. However, there is still much work to be done.

Acknowledging that when we model we always lose essential details, George Box said “All models are false but some models are useful.”  At some point we all have to decide to answer the wrong question, whether in the economy or just in deciding when to go to bed.  Otherwise we wouldn’t be able to get up in the morning.