Less doubt, more data - How to set up a lightning-fast data development cycle

Less doubt, more data - How to set up a lightning-fast data development cycle

‍

In this blog post, I’ll show you how to use our CLI and GitHub Actions for better data development in Lightdash - getting you closer to your Lightdash project essentially building itself 💪

‍

“the developer experience for data teams really sucks”

This was the overall feeling we got from talking to our community of data practitioners.

The day-to-day development in data is full of doubt, insecurity and leaps of faith. There’s the constant switching between applications and praying that everything works along the way: trying to validate your SQL, opening a PR and hoping it doesn’t break anything in your BI tool, and syncing all of your changes across your data pipeline.

And if something breaks? well… you need to start the same process all over again.

his developer experience is unproductive but familiar to everyone trying to develop in data. A small mistake can cost you multiple hours and immeasurable frustration.

So, we decided to try to do something about this at Lightdash and took on a big goal:

"Improving the data developer workflow"

There are two strategies that can help improve confidence and productivity in your data development workflows that we’ve borrowed from the world of software engineering:

• Automation: The more automation you introduce, the fewer interruptions you get during the day and the more you eliminate human error
• Testing: The better your testing tools and testing culture, the sooner and more effortlessly bugs are found

‍

Based on these 2 strategies, we recently built a bunch of new features in Lightdash - and that’s what I’m here to talk about 😊

‍

Here are some tips to get you developing faster and more reliably with Lightdash

‍

Auto-generate your schema.yml files

There isn’t a more tedious job than manually typing all the tables and their dimensions in the schema.yml. Luckily for you, these can be auto-generated with the Lightdash CLI tool! Below, you can see the most common use cases but check out all the options in our docs here

# Lightdash CLI command
lightdash generate # all models
lightdash generate -s payments  # just payments
lightdash generate -s payments+ # payments and all children
lightdash generate -s +payments # payments and all parents
lightdash generate -s +payments+ # payments and all children and parents
lightdash generate -s tag:prod # all models with the prod tag
lightdash generate -s payments customers # two specific models
lightdash generate -s payments+ +customers tag:prod # mix and match

‍

Create projects from scratch, in one line

We know the feeling of long onboarding forms for your BI tool…because we used to have one 😅

When you’re faced with a form of complicated configurations you need before you can even get started, it can be a bit of a bummer.

So, we decided to make this as easy and automated as possible for our users, and we went from this ...

… to this!

# Lightdash CLI command
lightdash deploy --create

If you are new to Lightdash you can see our docs on how to create your first project in Lightdash.

‍

Test your changes in Lightdash before you share them

Avoid the “these numbers look wrong…” face-palm moment and test your changes before adding them to production.

You can create preview environments in Lightdash that self-destruct once you’re done with them. They also hot reload with every dbt compile making development a breeze.

You can create charts and dashboards using your new metrics and even share them with your colleagues to confirm your results.

# Lightdash CLI command
lightdash preview

‍

Test pull requests

With Lightdash Previews, you can now do full stack reviews of Pull Requests before merging. You can hook up Lightdash Previews to GitHub Actions so that you can automatically create a test environment for every PR where any reviewer can test the changes before merging to production.If you're using GitHub you can read our docs on how to set up developer previews on your pull requests.

This can also be automated in other CI/CD pipelines by using the start and stop preview commands.

# Lightdash CLI commands
lightdash start-preview --name PR_123 # starts or updates a preview project
lightdash stop-preview --name PR_123 # stops/destroy preview project

Keep Lightdash in sync

You shouldn’t have to worry that your business users are using the latest metric definition or seeing up-to-date data. Once you merge changes into production, they should just sync everywhere else.

In Lightdash, we’ve made it simple to automate this syncing process and rest easy knowing that we’re always up to date with the latest changes to your dbt project 🧘 That means once you hit merge to main, Lightdash will update to show the latest version of your dbt project.

You can do this now by setting up a simple GitHub Action using this template and you can read the full instructions on how to do this in our docs here.

‍

Now, go forth and push for that 4 day work week (we know you’ve just become way more productive)

At Lightdash, we’re trying to make data development great. Automating tedious tasks and providing a quick way to test will not only increase productivity but also keep your entire team sane.

These features are just the tip of the iceberg when it comes to improving these workflows - going forward we’re taking on an even bigger mission with our new plan to make developing data in Lightdash a joyful experience. Because data teams deserve better!

If you agree (or disagree), we’d love to hear about it. Join the conversation in our Slack Community and let us know what you think!