In this tutorial, I’m going to introduce you to BI-as-code with a series of recipes using our new Lightdash CLI. By the end of this tutorial you’ll understand best practice for managing your BI code including:
If you want to follow along with the tutorial be sure to setup your own Lightdash project and install the CLI tool.
Our aim with these tutorials is to help you build a best-in-class developer experience, which will make developing your analytics code a joy and increase your team velocity by shipping quality code as fast as possible. Now let’s get started and level up together!
In Lightdash, everything you need for BI is written as code in your dbt project. The best part is that you’ve probably already written 80% of it. So let’s finish that last 20%:
After running our dbt project, lightdash generate is going to auto-generate all the configuration that Lightdash needs to build your semantic layer. Here’s an example of an auto-generated .yml file that you might see:
The customers.yml file tells Lightdash about all the attributes of customers. So by now our Lightdash project usually looks a little like this:
Now let’s push this project to Lightdash so we can start exploring our data through the UI.
To deploy your project simply use lightdash deploy
And that’s it! Head to the link and you’ll be able to start building charts off your customers model in dbt. And it took about a minute!
Deploying straight to production is fast but often when we’re developing our BI projects (altering metrics, adding new attributes, changing charts) we want to preview those changes before we push them to everyone else.
Imagine we’ve altered our customers.yml file to add a new metric:
Before pushing this out to our organization we certainly want to know that it works. What if we wrote the .yml file incorrectly or our metric definition is wrong?
Developer previews are temporary Lightdash projects completely separated from your production project. To start a preview run:
And that’s it! Your preview is a fully functional Lightdash project and when you’re finished, everything is torn down, so you know any changes you make won’t affect production. When you’re happy with your changes just push them to your production project:
Automating the deployment of our code is essential for governing our production environment. It allows us to add restrictions like:
By automating the deployment we also reduce the risk of human error during deploys! Thank you robots!
I’m going to use Github Actions to build our pipeline but you could apply these principals to your favourite CI/CD tool (CircleCI, Travis, …). To create our pipeline, we’ll start by creating a workflow file in our repository:
Our workflow provides a set of steps to carry out on our Lightdash project, but it only triggers when we commit our Lightdash project to our main branch. That way we can protect the main branch and only allow code to be merged into main after a thorough review. By just representing our BI-as-code we’ve automatically got all the governance features that come with code review!
Now when we merge to code into main we’ll see the action fire and our Lightdash project will update! And although we’ve introduced restrictions for deploying to production, there’s nothing standing in the way of us continuing to use lightdash preview on our laptops to get a lightning fast preview of our changes before we commit them to main.
And that’s it! If you followed along until this point, you’ve now got a best-practice setup for both development and production:
I hope that’s just enough to get you started with the Lightdash CLI and a glimpse into the power of managing your BI-as-code with Lightdash. Now open your terminal and get to work!