Many organizations design metrics incorrectly. As such, they get little, no, or negative value from those metrics. Unfortunately this means that the enormous amount of money spent by organizations on building data infrastructure and the care, dedication, and craft of data engineering teams, are often wasted. This is the ‘last-mile' problem of data and it has yet to be solved.
In this article, we'll share some high-level guidelines for building new metrics and to help you evaluate the efficiency of your existing metrics.
The problem: Misunderstanding of metrics
Companies can misuse metrics in many ways. Here are some examples we’ve seen throughout our careers:
- Metrics that don’t measure what stakeholders think they measure
- Metrics that are too complicated for stakeholders to understand
- Metrics that are too complicated for stakeholders to trust
- Too many metrics
- “Vanity metrics” that look good, but do not actually relate to an important process or outcome
- “Metrics isolate” that do not have a causal relationship with any important business process. Also known as "The Abstract Art of Metrics"
- “Artifact metrics” were useful at one point, but they no longer abstract relevant information
The solution: Use these design principles when building metrics
To avoid falling into the trap of misunderstanding and ensure that you're building impactful metrics for your organization, it's best to use some guidelines.
When identifying metrics, remember that metrics should be:
Measurable
A metric must be measurable throughout the entire time period it serves a business function.
Actionable
A metric should be designed so that stakeholders can easily understand and act upon the information it contains. Metrics should be as simple as possible, but no simpler. Too often, metric designs are overly complex. This leads stakeholders to distrust the information.
Metrics also should contain information that is directly relevant to business processing. Companies can make thousands of metrics that are “interesting,” but do not help stakeholders make decisions. This is bad. If there are too many metrics, decision-making processes become confusing and misleading. Better too few than too many metrics.
Causal
When possible, metrics should be based on causal relationships. Metrics are not artifacts that sit on a dashboard and look pretty. They are supposed to guide and inform decision-making. If a metric goes in a particular direction, it should cause a particular outcome. If a metric goes up, and it is supposed to cause an outcome to go up, but the outcome doesn’t do that, the metric is not useful. Similarly, if both a metric and an outcome go up, but they are only correlated, this can mislead teams’ decision-making processes.
Causal inference is not easy to determine. It could be thought of as a scale with correlation on one side and correlation on the other. Teams should try to push the scale towards the causal side through an accumulation of evidence. Causal relationships of metrics should be the goal, even if it is often difficult.
Iterable
Metrics are not static or sacred. As the world changes, the data artifacts created by the processes will change too. Some of that may become less useful. Similarly, as a business changes, business goals will change, which will require a different set of metrics.
Teams should deprecate and change metrics to reflect these changes without ego. A company should not have a preset goal of “five most important metrics,” when in fact only four may be useful.
Customizing metrics to your organization
These are high-level guidelines—and you might even have more to add to this list. At Transform, we work with your organization to determine the most effective way to track business metrics so that they align with your organizational goals.
Transform’s mission is to “make data accessible.” We build interfaces that allow anyone to reliably access data. We believe that metrics are the unit of abstraction from these data tools which will most often inform decision-making processes. Transform ensures that organizations can make and access metrics consistently. See Transform in action.
