Get started with BI-as-code

Get started with BI-as-code

In this tutorial, I’m going to introduce you to BI-as-code with a series of recipes using our new Lightdash CLI. By the end of this tutorial you’ll understand best practice for managing your BI code including:

• How to deploy to production from the CLI
• How to preview your changes before deploying
• How to automate production deployments with Github Actions
  ‍

If you want to follow along with the tutorial be sure to setup your own Lightdash project and install the CLI tool.

Our aim with these tutorials is to help you build a best-in-class developer experience, which will make developing your analytics code a joy and increase your team velocity by shipping quality code as fast as possible. Now let’s get started and level up together!

‍

An instant BI platform from your dbt project

In Lightdash, everything you need for BI is written as code in your dbt project. The best part is that you’ve probably already written 80% of it. So let’s finish that last 20%:

# Run our dbt project to create our analysis-ready tables in our data warehouse
dbt run

# Auto-generate our Lightdash configuration
lightdash generate

‍

After running our dbt project, lightdash generate is going to auto-generate all the configuration that Lightdash needs to build your semantic layer. Here’s an example of an auto-generated .yml file that you might see:

# customers.yml
version: 2
models:
  - name: customers
    columns:
      - name: customer_id
        meta:
          dimension:
            type: string
      - name: name
        meta:
          dimension: 
            type: string
      - name: created_at
        meta:
          dimension:
            type: timestamp

‍

The customers.yml file tells Lightdash about all the attributes of customers. So by now our Lightdash project usually looks a little like this:

lightdash-project
├── dbt_project.yml
└── models
    ├── customers.sql
    └── customers.yml

Now let’s push this project to Lightdash so we can start exploring our data through the UI.

‍

Level 1: Deploy directly to production

To deploy your project simply use lightdash deploy

lightdash login
lightdash deploy

# Your Lightdash project is ready at:
#    https://app.lightdash.cloud/projects/...

And that’s it! Head to the link and you’ll be able to start building charts off your customers model in dbt. And it took about a minute!

Deploying straight to production is fast but often when we’re developing our BI projects (altering metrics, adding new attributes, changing charts) we want to preview those changes before we push them to everyone else.

‍

Level 2: Preview changes in a dev environment

Imagine we’ve altered our customers.yml file to add a new metric:

# customers.yml
...
models:
  - name: customers
    columns:
      - name: customer_id
        meta:
          metrics:
            customer_count: 
              type: count_distinct
...

Before pushing this out to our organization we certainly want to know that it works. What if we wrote the .yml file incorrectly or our metric definition is wrong?

Developer previews are temporary Lightdash projects completely separated from your production project. To start a preview run:

lightdash preview

# Your Lightdash preview is ready at:
#    https://app.lightdash.cloud/projects/...

And that’s it! Your preview is a fully functional Lightdash project and when you’re finished, everything is torn down, so you know any changes you make won’t affect production. When you’re happy with your changes just push them to your production project:

lightdash deploy

‍

Level 3: Deploy with GitHub Actions

Automating the deployment of our code is essential for governing our production environment. It allows us to add restrictions like:

• Code must be committed to version control to be deployed to production
• Code must be tested before it can be deployed

By automating the deployment we also reduce the risk of human error during deploys! Thank you robots!

I’m going to use Github Actions to build our pipeline but you could apply these principals to your favourite CI/CD tool (CircleCI, Travis, …). To create our pipeline, we’ll start by creating a workflow file in our repository:

lightdash-project
├── .github
│   └── workflows
│       └── deploy.yml
├── dbt_project.yml
└── models
    ├── customers.sql
    └── customers.yml

Our workflow provides a set of steps to carry out on our Lightdash project, but it only triggers when we commit our Lightdash project to our main branch. That way we can protect the main branch and only allow code to be merged into main after a thorough review. By just representing our BI-as-code we’ve automatically got all the governance features that come with code review!

# ./github/workflows/deploy.yml
name: lightdash-deploy

on:
  # Triggers workflow when we merge code to main
  push:
    branches: [ "main" ]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v1

      - name: Install dbt dependencies
        run: |
          pip install dbt-core dbt-bigquery
          dbt deps

      - name: Install Lightdash CLI
        run: npm i -g @lightdash/cli

      - name: dbt run
        run: dbt run
      
      - name: lightdash deploy
        run: lightdash deploy
        
    env:
      LIGHTDASH_URL: ${{ secrets.LIGHTDASH_URL }}
      LIGHTDASH_PROJECT: ${{ secrets.LIGHTDASH_PROJECT }}
      LIGHTDASH_API_KEY: ${{ secrets.LIGHTDASH_API_KEY }}

Now when we merge to code into main we’ll see the action fire and our Lightdash project will update! And although we’ve introduced restrictions for deploying to production, there’s nothing standing in the way of us continuing to use lightdash preview on our laptops to get a lightning fast preview of our changes before we commit them to main.

‍

Conclusion

And that’s it! If you followed along until this point, you’ve now got a best-practice setup for both development and production:

• We’ve made our development cycle 10x faster by allowing you to instantly preview changes as you write them. This shortens the build-and-test loop while you hack around in a safe sandbox.
• We can now deploy to production with confidence! By using automation and code review, we can be confident that our new code is ready to ship to the company, no more broken dashboards. Once you feel confident with your automated deployment you’ll find you deploy more frequently and with few errors, making your overall time to production substantially faster!

I hope that’s just enough to get you started with the Lightdash CLI and a glimpse into the power of managing your BI-as-code with Lightdash. Now open your terminal and get to work!