When considering internal data or the results of a study, often business leaders either take the evidence presented as gospel or dismiss it altogether. Both approaches are misguided. What leaders need to do instead is conduct rigorous discussions that assess any findings and whether they apply to the situation in question.
Such conversations should explore the internal validity of any analysis (whether it accurately answers the question) as well as its external validity (the extent to which results can be generalized from one context to another). To avoid missteps, you need to separate causation from correlation and control for confounding factors. You should examine the sample size and setting of the research and the period over which it was conducted. You must ensure that you’re measuring an outcome that really matters instead of one that is simply easy to measure. And you need to look for—or undertake—other research that might confirm or contradict the evidence.
By employing a systematic approach to the collection and interpretation of information, you can more effectively reap the benefits of the ever-increasing mountain of external and internal data and make better decisions.
When managers are presented with internal data or an external study, all too often they either automatically accept its accuracy and relevance to their business or dismiss it out of hand.
Leaders mistakenly conflate causation with correlation, underestimate the importance of sample size, focus on the wrong outcomes, misjudge generalizability, or overweight a specific result.
Leaders should ask probing questions about the evidence in a rigorous discussion about its usefulness. They should create a psychologically safe environment so that participants will feel comfortable offering diverse points of view.
Let’s say you’re leading a meeting about the hourly pay of your company’s warehouse employees. For several years it has automatically been increased by small amounts to keep up with inflation. Citing a study of a large company that found that higher pay improved productivity so much that it boosted profits, someone on your team advocates for a different approach: a substantial raise of $2 an hour for all workers in the warehouse. What would you do?
