Lightdash x dbt Semantic Layer

‍

As a dbt Semantic Layer Launch Partner, we’re so excited to share what we’ve been cooking up for you at Lightdash! Before we dive into the integration - let’s take a look at how the dbt Semantic Layer works and why we chose to integrate it into Lightdash. The integration is live right now for all Lightdash users - here's the docs page on getting started! 

‍

Open standards and a single source of truth

‍

The much-anticipated dbt Semantic Layer is here! With the launch of the Semantic Layer, dbt Cloud users now have the power to define their core business metrics in one place and consume them in many downstream tools (including Lightdash of course).

This is a huge deal. The data community has been pushing for an open standard for metrics that would finally allow interoperability between data tools; freeing metrics from proprietary tools where they are commonly found today. This resonates deeply with the core of Lightdash, which has always been built from day one to be open and integrate with the tools that data teams already use and love. 

So now if you’re using dbt metrics, whether you're in Lightdash or a Python notebook, you can pull "revenue for the last quarter" and... you know for certain you’ll have the same numbers in both places, all coming from the same source!

‍

The Semantic Layer makes your dbt project more flexible

‍

dbt's Semantic Layer overcomes a thorny problem with traditional data modelling in dbt. Let's say you have a table of users and your stakeholders ask for some daily statistics. No problem, you create a new model `users_daily` and enjoy all the productivity delights of SQL+jinja and check-in your new model into version control. Oh to be an analytics engineer! Then your stakeholders ask for the statistics on a weekly granularity and you build another model called `users_weekly` - what if you want user statistics by geography or acquisition channel? 

‍

With the dbt Semantic Layer, you retain a great deal more flexibility. Instead of giving your end users pre-aggregated tables, we can take our base user table and define our user statistics as dbt metrics, and define all the different ways they can be aggregated (by day, by week, by geography). Our stakeholders are now free to choose any statistic they like and cut it any way they like, without us having to build a new model. This is really powerful, and to be able to do it all while not having to write new models or create new siloes of data logic in separate tools is what we’re excited for Lightdash users to take advantage of! 

‍

So while dbt modelling gave analysts huge productivity boosts in creating fixed models, the dbt Semantic Layer now gives us incredible flexibility by empowering data consumers to easily access analytics cut to any shape they like from the tools they already love (like Lightdash and the other Semantic Layer partners). And all of that without sacrificing governance and our precious dbt macros.

‍

You need to see it to feel it! With that, here are 3 simple steps for dbt Cloud users to create their first metric and visualise it in Lightdash.

‍

3 steps to get started with the dbt Semantic Layer & Lightdash

‍

Step 1: define your metric as yaml

‍

The first step is to define a metric. In this case, we’re going to take our table of users and we’re going to define the “Total unique users” metric, which for our business is the count of distinct user ids:

‍

metrics: 
  - name: total_unique_users
    label: Total unique users
    model: ref('users')
    type: count_distinct
    expression: user_id
    timestamp: created_at
    time_grains: [day, week, month]
    dimensions:
      - is_active
      - subscription_type

‍

Notice how we’re using tables, columns, and SQL functions (like count distinct) that are technical terms to define the business term “total unique users”. This is the semantic part! Now hit compile and run your project in dbt Cloud.

‍

Step 2: Connect to the dbt Semantic Layer

‍

In Lightdash setting up the integration is really simple. You just need 3 things from your dbt Cloud project:

• The project id
• The job id (the one you just ran above with your metrics)
• And an API key

‍

Head on over to the settings page and click “Connect dbt Semantic Layer” to get started. Once the connection is set up, you’ll immediately have access to your metrics queryable in Lightdash. 

‍

‍

Step 3: Visualise your metrics

‍

Now let’s bring our metrics to life. In Lightdash head over to the SQL console and you’ll see a brand new sidebar showing up with your metrics. Click any metric, hit run and start building and sharing charts immediately. 

‍

‍

Conclusion

We’re only just getting started with our plans to build out even more functionality for the dbt Semantic Layer, but we’re excited that folks can already explore, analyse, visualise and share their dbt metrics natively today in Lightdash. Over the coming months, we’ll be releasing even more features on top of this, make sure to check out our public roadmap to see what’s coming next! 

Lightdash started out with a simple idea: to locate business critical metrics in a single place alongside your dbt transformations. With the meteoric rise of dbt, the community finally has a real shot at a single interoperable layer that can serve metrics to multiple tools.

The dbt Semantic Layer integration is already available in Lightdash for you to use and try out. Drop us a line in the community if you need more help getting started and share your feedback on our newest integration!