Introducing dbt + Transform: Bring data transformations into your metric store

If you’re a data practitioner, you’ve probably heard of dbt. dbt is a data transformation tool where data analysts and engineers can transform, test, and document data in the data warehouse.

We see data modeling and transformation as essential practices that happen upstream of a metric store. But it is critical that all of the information and context around those transformations is readily accessible in your metric store.

We’re excited to introduce our first integration with dbt. This integration gives you the power to define Transform data sources using dbt models. Transform then surfaces useful metadata from your dbt models—like the time of the last successful run—and links to model documentation directly in the Transform user interface (UI).

How to use Transform and dbt together

To use this integration, you’ll have to complete a few simple setup steps. You can learn more about the integration setup in our product documentation.

Using dbt models for your Transform metrics

Once you complete the setup instructions, you’re ready to add your dbt models as a data source in your Transform source configuration files.

Add your dbt model in the dbt_model parameter.

---
data_source:
  name: calls
  description: This data source is sourced from the demo.calls table. Each row in this table represents a call. Each call has a user_id representing the user making that call.
  owners:
    - larry+david@transformdata.io
  sql_query: select * from demo_dbt.calls_base
  dbt_model: demo_dbt.calls_base

  identifiers:
    - name: call
      type: primary
      expr: call_id
    - name: user_id
      type: foreign

  measures:
    - name: calls
      description: The total number of calls made.
      expr: 1
      agg: sum

  dimensions:
    - name: ds
      type: time
      is_partition: false
      type_params:
        is_primary: True
        time_format: YYYY-MM-DD
        time_granularity: day

  mutability:
    type: immutable

Viewing dbt source freshness in Transform Metric Store

You’ll now be able to see the source freshness for the dbt models you use as data sources in the Transform UI. In the UI, navigate to a metric page where you used a dbt model as a data source.

There are two places in the UI where you can view dbt source freshness information:

1. Click the “Metric Lineage” button in the Metric Definition section.

2. Click the “dbt Source Freshness” button in the Metric Definition section.

Viewing dbt lineage docs in Transform Metric Store

You’ll now also be able to access the lineage docs link for every dbt model that you use as a data source in Transform. Follow these steps:

• Click the “Metric Lineage” button.
• Click on any Data Source card in the lineage graph with a dbt icon.
• Scroll down and click the “dbt Source File” button.

Alternatively:

• Click the “dbt Source Freshness” button.
• Click on the data source name of any of the Data Source cards.

Looking ahead

This is just the first step. We’re working to bring in more ways to surface useful data from your upstream tools to create a more robust metric store.

We plan on building additional functionality with dbt to bring even more power to this integration. For example, we’re adding functionality that auto-runs Transform materializations triggered by dbt job completions—which will reduce the amount of manual scheduling for materializations.

We’re looking forward to building out more functionality with dbt to create the best experience for our customers.