I wrote before about types of value generated by data science projects. What happens if you can't report value in dollars?

James Q. WIlson classified bureaucratic agencies based on whether their inputs and outputs were measurable.

Wilson's typology of agencies

		Inputs
		measurable	not measurable
Outputs	measurable	"Production" IRS; Social Security Administration	"Craft" wartime military; police
	not measurable	"Procedural" regulatory agencies; peacetime military	"Coping" State Department; teaching

The IRS is a production agency because we can measure its inputs (number of returns processed, number of audits conducted) and outputs (revenue collected). SSA is the same: inputs checks issued and benefits paid, an output is lower senior poverty.

The wartime military is a craft agency because it is impossible, given the fog of war, to fully account for effort expended during battle, but you can tell whether the platoon took that hill. Police work is similar: it's difficult to measure the effort of walking the beat, but we can measure the crime rate.

The peacetime military is interestingly the opposite, a procedural agency. We can document the specific training and exercises conducted, but how do you gauge the deterrent effect of a strong standing force when there's no shooting? Regulatory agencies are similar: we can see the regulations created, but the deterrent effects are impossible to fully quantify. What are the monetary benefits of reduced air pollution?

Coping agencies are particularly interesting because neither inputs nor outputs can be quantified. Consider a diplomat in the State Department. Is their effectiveness to be predicted by the length of their briefing paper? How much did that one paper decrease the likelihood of a trade war, and what would that trade war have cost the United States? With teaching, how do you measure the inputs (quality of instruction)? For outputs, we can use test scores, but those are affected more by household income and prior achievement than the current instruction. Plus, how do you measure intangible outputs like socialization?

If you acknowledge that the inputs of these agencies are costly and the outputs are valuable, then it's reasonable to consider the relative value of these agencies. Should we invest in the Environmental Protection Agency (to help address climate change) or the State Department? Which costs more or addresses the bigger problem?

Efficiency (aka ROI) is defined as:

$$\mathrm{efficiency} = \mathrm{ROI} = \frac{\mathrm{outputs}}{\mathrm{inputs}}$$

Suppose the inputs, outputs, or both aren't measurable. Then you don't have what you need to plug into this formula.

Example: Staffing

You invest in a data engineer to partner with your data science team. You can measure their salary, but how do you quantify the increased satisfaction of your data scientists who can then focus on modeling instead of pipelines? This may decrease attrition among the data scientists. Does retention matter, and if so, how much?

Your team builds a chatbot to let customers conduct business. Your KPI is a reduction in costs by reducing the number of agents required to assist customers. You're surprised that the reduction in costs barely covers the cost of the team building the chatbot. You measured inputs and outputs, so the ROI must be well-defined, right?

Missing inputs

To take on the chatbot project, you delayed data science work on GDPR compliance. This delay kept the risk of penalties for non-compliance high. You might estimate the opportunity cost like:

$$(\text{0.1% marginal probability of being sued}) \times (\text{\$10M typical penalty for your industry})$$

Are you sure about that 0.1%? How about that $10M? This estimate of opportunity cost could vary 100x given small changes in those guesses.

Missing outputs

Your chatbot works! Then you hear from marketing about surveys that show:

1. Your target demographic thinks tech-forward companies are more desirable.
2. Prospective customers visiting your site see the chatbot and click to pages giving details.
3. Current customers love it!

It seems like this should increase acquisition and decrease churn, both valuable. How would you measure that value? You might be able to get a sense of #3, but a clean A/B test would be difficult. It's hard to gauge #1 and #2, as marketers know, because you can't prevent your control group from learning about your chatbot elsewhere. This means that you are certain to misvalue the chatbot.

---

Data science projects often have value that can't be quantified.