If you’re a data practitioner, you’ve probably heard of dbt. dbt is a data transformation tool where data analysts and engineers can transform, test, and document data in the data warehouse.
We see data modeling and transformation as essential practices that happen upstream of a metric store. But it is critical that all of the information and context around those transformations is readily accessible in your metric store.
We’re excited to introduce our first integration with dbt. This integration gives you the power to define Transform data sources using dbt models. Transform then surfaces useful metadata from your dbt models—like the time of the last successful run—and links to model documentation directly in the Transform user interface (UI).
How to use Transform and dbt together
To use this integration, you’ll have to complete a few simple setup steps. You can learn more about the integration setup in our product documentation.
Using dbt models for your Transform metrics
Once you complete the setup instructions, you’re ready to add your dbt models as a data source in your Transform source configuration files.
Add your dbt model in the dbt_model parameter.
--- data_source: name: calls description: This data source is sourced from the demo.calls table. Each row in this table represents a call. Each call has a user_id representing the user making that call. owners: - firstname.lastname@example.org sql_query: select * from demo_dbt.calls_base dbt_model: demo_dbt.calls_base identifiers: - name: call type: primary expr: call_id - name: user_id type: foreign measures: - name: calls description: The total number of calls made. expr: 1 agg: sum dimensions: - name: ds type: time is_partition: false type_params: is_primary: True time_format: YYYY-MM-DD time_granularity: day mutability: type: immutable
Viewing dbt source freshness in Transform Metric Store
You’ll now be able to see the source freshness for the dbt models you use as data sources in the Transform UI. In the UI, navigate to a metric page where you used a dbt model as a data source.
There are two places in the UI where you can view dbt source freshness information:
- Click the “Metric Lineage” button in the Metric Definition section.
2. Click the “dbt Source Freshness” button in the Metric Definition section.
Viewing dbt lineage docs in Transform Metric Store
You’ll now also be able to access the lineage docs link for every dbt model that you use as a data source in Transform. Follow these steps:
- Click the “Metric Lineage” button.
- Click on any Data Source card in the lineage graph with a dbt icon.
- Scroll down and click the “dbt Source File” button.
- Click the “dbt Source Freshness” button.
- Click on the data source name of any of the Data Source cards.
This is just the first step. We’re working to bring in more ways to surface useful data from your upstream tools to create a more robust metric store.
We plan on building additional functionality with dbt to bring even more power to this integration. For example, we’re adding functionality that auto-runs Transform materializations triggered by dbt job completions—which will reduce the amount of manual scheduling for materializations.
We’re looking forward to building out more functionality with dbt to create the best experience for our customers.