{"id":348,"date":"2025-06-22T23:47:42","date_gmt":"2025-06-22T23:47:42","guid":{"rendered":"https:\/\/haptonstahl.org\/polimath\/?p=348"},"modified":"2025-06-22T23:47:44","modified_gmt":"2025-06-22T23:47:44","slug":"one-number-doesnt-do-it-context-matters","status":"publish","type":"post","link":"https:\/\/haptonstahl.org\/polimath\/one-number-doesnt-do-it-context-matters\/","title":{"rendered":"One number doesn’t do it: Context matters"},"content":{"rendered":"\n
<\/blockquote>\n\n\n\n

“Our net promoter score for the new version is 42.”<\/em><\/strong><\/p>\n\n\n\n

Okay. So what? Is 42 good or bad? Should we continue rolling out the new version, or should we roll back to the previous version?<\/p>\n\n\n\n

“The previous version had an NPS of 35.”<\/em><\/strong><\/p>\n\n\n\n

Ah, some context! It looks<\/em> like the new version is an improvement.<\/p>\n\n\n\n

Wait, are we sure?<\/p>\n\n\n\n

“We surveyed 10 people.”<\/em><\/strong><\/p>\n\n\n\n

Hmm. That would be a no.<\/p>\n\n\n\n

\n\n\n\n

When you report a number, provide a quantitative context.<\/em><\/p>\n\n\n\n

Usually this means:<\/p>\n\n\n\n

    \n
  1. Comparison<\/strong>: A second number (or more) that the audience is familiar with and to which they can compare the first number.<\/li>\n\n\n\n
  2. Precision<\/strong>: Is there uncertainty about the numbers? How much?<\/li>\n\n\n\n
  3. Significance<\/strong>: A sense of how much weight they should put on the difference between the two numbers when making a decision<\/strong>.<\/li>\n<\/ol>\n\n\n\n

    Note that precision is not the same as significance.<\/p>\n\n\n\n

      \n
    • Precision<\/strong>: Literally, think of this as the number of decimal places. Ways to describe this include statistical significance, standard errors, frequentist confidence intervals, Bayesian credible intervals, bootstrapped intervals, hypothesis tests, prediction intervals, or just a count of data points.<\/li>\n\n\n\n
    • Significance<\/strong>: Literally, is this difference important? <\/em>This is not <\/em>the same as statistical significance! This is a business determination, not a measurement.<\/li>\n<\/ul>\n\n\n\n

      Let’s look at a simple example.<\/p>\n\n\n\n

      \"\"<\/figure>\n\n\n\n

      Is 86% good or bad? Does this mean our retention efforts are working? We could add a comparison number explicitly by saying the industry average is Y% or last year we had 80%. A good choice is the percent difference with a familiar value, like last year’s retention.<\/p>\n\n\n\n

      \"\"<\/figure>\n\n\n\n

      This is better. Now we know that retention went up. Or did it? How precise are these numbers? One way to convey this would be to give a confidence or prediction interval, but these are not based on samples, so there is no sampling uncertainty<\/strong>. <\/strong>A sufficient way to convey precision is to say how many employees there are.<\/p>\n\n\n\n

      \"\"<\/figure>\n\n\n\n

      In this scenario, we retained 6\/7 employees this year and 4\/5 last year. Attrition was one person each year. There’s no real change, so let’s say so by stating the substantive significance<\/strong>.<\/p>\n\n\n\n

      \"\"<\/figure>\n\n\n\n

      This is good. We present the retention, a comparison, a sense of the precision, and the significance.<\/p>\n\n\n\n

      Now, suppose we had the same retention rate and percent change, but a lot more employees.<\/p>\n\n\n\n

      \"\"<\/figure>\n\n\n\n

      We lost 1000 employees, but the long-term average tenure is 7 years per employee, just like the previous scenario, and it went up by two years (long-term average). We have a lot more data, so we are more precise: the 6% increase is not<\/strong> a quirk in the data. This is good news — we should make sure the audience knows that.<\/p>\n\n\n\n

      \"\"<\/figure>\n\n\n\n

      Note that the number we’re reporting (86%) is the same for the two scenarios, but the takeaway is very different because of the context.<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"

      “Our net promoter score for the new version is 42.” Okay. So what? Is 42 good or bad? Should we continue rolling out the new version, or should we roll back to the previous version? “The previous version had an NPS of 35.” Ah, some context! It looks like the new version is an improvement. […]<\/p>\n","protected":false},"author":2,"featured_media":398,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"nf_dc_page":"","pagelayer_contact_templates":[],"_pagelayer_content":"","footnotes":""},"categories":[6],"tags":[8,13],"class_list":["post-348","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-science","tag-measurement","tag-uncertainty"],"_links":{"self":[{"href":"https:\/\/haptonstahl.org\/polimath\/wp-json\/wp\/v2\/posts\/348","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/haptonstahl.org\/polimath\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/haptonstahl.org\/polimath\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/haptonstahl.org\/polimath\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/haptonstahl.org\/polimath\/wp-json\/wp\/v2\/comments?post=348"}],"version-history":[{"count":5,"href":"https:\/\/haptonstahl.org\/polimath\/wp-json\/wp\/v2\/posts\/348\/revisions"}],"predecessor-version":[{"id":408,"href":"https:\/\/haptonstahl.org\/polimath\/wp-json\/wp\/v2\/posts\/348\/revisions\/408"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/haptonstahl.org\/polimath\/wp-json\/wp\/v2\/media\/398"}],"wp:attachment":[{"href":"https:\/\/haptonstahl.org\/polimath\/wp-json\/wp\/v2\/media?parent=348"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/haptonstahl.org\/polimath\/wp-json\/wp\/v2\/categories?post=348"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/haptonstahl.org\/polimath\/wp-json\/wp\/v2\/tags?post=348"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}