Metrics Design Guide

Transform’s mission is to “make data accessible.” We build interfaces that allow anyone to reliably access data. We believe that metrics are the unit of abstraction from these data tools which will most often inform decision-making processes. Transform ensures that organizations can make and access metrics consistently. This is a hard technical problem we are trying to solve. However, even if Transform can do this, users face another challenge.

Many organizations design metrics incorrectly. As such, they get little, no, or negative value from those metrics. The enormous amount of money spent by organizations on building data infrastructure and the care, dedication, and craft of data engineering teams, are therefore often wasted. This is the ‘last-mile' problem of data, and it has yet to be solved!

The problem: misunderstanding of metrics

Companies can misuse metrics in many ways. Here are some examples we’ve seen throughout our careers:

  • Metrics that don’t measure what stakeholders think they measure
  • Metrics that are too complicated for stakeholders to understand
  • Metrics that are too complicated for stakeholders to trust
  • Too many metrics
  • “Vanity metrics” that look good, but do not actually relate to an important process or outcome
  • “Metrics isolate” that do not have a causal relationship with any important business process. Also known as The Abstract Art of Metrics
  • “Artifact metrics” were useful at one point, but they no longer abstract relevant information

The solution: the Transform Metrics Design Guide

Measurable
A metric must be measurable throughout the entire time period it serves a business function.

Actionable
A metric should be designed so that stakeholders can easily understand and act upon the information it contains. Metrics should be as simple as possible, but no simpler. Too often, metric designs are overly complex. This leads stakeholders to distrust the information.

Metrics also should contain information that is directly relevant to business processing. Companies can make thousands of metrics that are “interesting,” but do not help stakeholders make decisions. This is bad. If there are too many metrics, decision-making processes become confusing and misleading. Better too few than too many metrics.

Causal
When possible, metrics should be based on causal relationships. Metrics are not artifacts that sit on a dashboard and look pretty. They are supposed to guide and inform decision-making. If a metric goes in a particular direction, it should cause a particular outcome. If a metric goes up, and it is supposed to cause an outcome to go up, but the outcome doesn’t do that, the metric is not useful. Similarly, if both a metric and an outcome go up, but they are only correlated, this can mislead teams’ decision-making processes.

Causal inference is not easy to determine. It could be thought of as a scale with correlation on one side and correlation on the other. Teams should try to push the scale towards the causal side through an accumulation of evidence. Causal relationships of metrics should be the goal, even if it is often difficult.

Iterable
Metrics are not static or sacred. As the world changes, the data artifacts created by the processes will change too. Some of that may become less useful. Similarly, as a business changes, business goals will change, which will require a different set of metrics.

Teams should deprecate and change metrics to reflect these changes without ego. A company should not have a preset goal of “five most important metrics,” when in fact only four may be useful.

Jacob White

Jacob White