MetricFlow: Your Powerful, Open Source Metrics Framework

Today is a big day—we’re proud to announce that MetricFlow (Transform’s metric creation framework) is now open source and available to all!

Starting today, you can download MetricFlow (`pip install metricflow`) and explore or contribute to the project on GitHub. We’re also starting the MetricFlow Slack community for collaborations, contributors, and general discussion on metric layers.

MetricFlow enables us to reach our extraordinarily ambitious goals to express our logic once and only once, in other words DRY. MetricFlow does this by working from direct and simple declarations of the metrics you care about, and dynamically constructing the complex logic to query those metrics across a range of related dimensions, time granularities, and other metrics. This MetricFlow logic is optimized into performant and legible SQL queries that run natively on your cloud warehouse.

MetricFlow is the culmination of so many ideas. We drew inspiration from a number of companies that have invested in metrics tooling, most notably from our teams’ experiences at Airbnb, and from semantic layers embedded in various data applications like Looker, PowerBI, and Tableau. While these tools have their own notable innovations, we believe that with broader inputs from the data community, it’s possible to achieve a loftier and more significant vision as a stand-alone metric layer—one where we are empowered instead of restrained by the endless possibilities (and questions) that come with data.

What is MetricFlow?

MetricFlow is a semantic layer and set of abstractions built around the powerful compute and storage offered by modern cloud data warehouses. The project is designed around the following core components:

Metric Spec: The specification encapsulates metric logic in a reusable and powerful set of abstractions: data_sources, measures, dimensions, identifiers, metrics, and materializations.
Metric Query Planner: MetricFlow creates a query plan using source data to construct metrics in the data warehouse. This includes warehouse-specific performance and legibility optimizations.
MetricFlow APIs: The CLI and Python APIs provide developer workflows, and enable users to build and execute metric querying logic.

With MetricFlow you can:

Construct metrics from powerful semantic models: At its core, MetricFlow is a semantic modeling framework. It constructs a relational graph of your underlying data model, giving you the power to construct and aggregate even your most complex metrics to dimensional granularities across your data warehouse. Powerful abstractions make this possible with less configuration than any other semantic framework.

Differences in code for differing MetricFlow use cases: finance, growth, and marketing.

Avoid code duplication and disjointed metric logic: MetricFlow uses a semantic model, generating SQL under the hood as you query your metrics. MetricFlow is optimized for the “DRY (Don’t Repeat Yourself)” principle to help you limit code duplication and therefore avoid disjointed metric logic.
Leverage optimized queries to any data warehouse: MetricFlow supports—and even optimizes—queries to any data warehouse. This is built on a flexible interface to define new database connectors and optimizations. This means metric definitions in MetricFlow will continually improve rendered SQL as more people contribute to the open source project.
Easily generate and maintain data marts in your warehouse: Metrics in MetricFlow, can easily be composed into data marts back to the warehouse. This reduces the need to redefine logic across multiple tables, and makes code updates as simple as editing a single line of code.
Seamlessly exchange metric models between tools: MetricFlow parses the YAML files that follow the MetricFlow spec into standardized Python objects that can easily be translated to and from a variety of metric specs. This allows you to seamlessly exchange metric models between tools in the data stack, reducing lock-in.

Graphic showing Python code for a financial analyst's MetricFlow use case.

Why open source?

We decided from the beginning that we wanted to build a foundation for MetricFlow and then open source our work. Historically, building metric layers has been a years-long process exclusive to the biggest data teams. MetricFlow sets a foundation for what we believe could be the most powerful semantic layer yet. We believe the only way to achieve our vision is to build MetricFlow in the open.

Three themes are top of mind for our team as we invest in this open source project:

Building together means we can be more ambitious: Building in the open means we benefit from not just our best ideas and our customers' feedback, but your best ideas and feedback as well. This creates a flywheel of value creation.
Providing greater transparency into your metric logic: MetricFlow’s spec, logic construction, and interfaces exist in the open. You can inspect and contribute to how the semantic layer renders and optimizes SQL queries.
Giving you the most flexibility: MetricFlow disentangles logic from applications, so you aren’t locked into vendors or tools. We believe core infrastructure and logic construction should never come with lock-in.

MetricFlow represents a significant step towards our vision to broaden access to data through metrics. We can’t wait to see how an open source framework will impact the way that people build metrics for their organizations.

The future of MetricFlow

MetricFlow is being released as v1.0 because we believe this is production-worthy software. Organizations, big and small, are relying on MetricFlow in production today. That being said, there is more work ahead. We hope you’ll contribute to Open Issues in MetricFlow.

In addition to MetricFlow, we will continue to build Transform, a Metric Store built on top of MetricFlow. Transform extends MetricFlow with:

Hosted MetricFlow Server capable of handling significant loads of queries with strong security protections
Caching mechanism native to your cloud data warehouse that enables more performant queries.
Integrations with the leading business intelligence tools and many other applications that consume metrics
Metric Catalog where you can accumulate and organize context around all of your metrics