In this blog post, I’ll show you how to use our CLI and GitHub Actions for better data development in Lightdash - getting you closer to your Lightdash project essentially building itself 💪
“the developer experience for data teams really sucks”
This was the overall feeling we got from talking to our community of data practitioners.
The day-to-day development in data is full of doubt, insecurity and leaps of faith. There’s the constant switching between applications and praying that everything works along the way: trying to validate your SQL, opening a PR and hoping it doesn’t break anything in your BI tool, and syncing all of your changes across your data pipeline.
And if something breaks? well… you need to start the same process all over again.
his developer experience is unproductive but familiar to everyone trying to develop in data. A small mistake can cost you multiple hours and immeasurable frustration.
So, we decided to try to do something about this at Lightdash and took on a big goal:
"Improving the data developer workflow"
There are two strategies that can help improve confidence and productivity in your data development workflows that we’ve borrowed from the world of software engineering:
Based on these 2 strategies, we recently built a bunch of new features in Lightdash - and that’s what I’m here to talk about 😊
There isn’t a more tedious job than manually typing all the tables and their dimensions in the schema.yml. Luckily for you, these can be auto-generated with the Lightdash CLI tool! Below, you can see the most common use cases but check out all the options in our docs here
We know the feeling of long onboarding forms for your BI tool…because we used to have one 😅
When you’re faced with a form of complicated configurations you need before you can even get started, it can be a bit of a bummer.
So, we decided to make this as easy and automated as possible for our users, and we went from this ...
… to this!
If you are new to Lightdash you can see our docs on how to create your first project in Lightdash.
Avoid the “these numbers look wrong…” face-palm moment and test your changes before adding them to production.
You can create preview environments in Lightdash that self-destruct once you’re done with them. They also hot reload with every dbt compile making development a breeze.
You can create charts and dashboards using your new metrics and even share them with your colleagues to confirm your results.
With Lightdash Previews, you can now do full stack reviews of Pull Requests before merging. You can hook up Lightdash Previews to GitHub Actions so that you can automatically create a test environment for every PR where any reviewer can test the changes before merging to production.If you're using GitHub you can read our docs on how to set up developer previews on your pull requests.
This can also be automated in other CI/CD pipelines by using the start and stop preview commands.
You shouldn’t have to worry that your business users are using the latest metric definition or seeing up-to-date data. Once you merge changes into production, they should just sync everywhere else.
In Lightdash, we’ve made it simple to automate this syncing process and rest easy knowing that we’re always up to date with the latest changes to your dbt project 🧘 That means once you hit merge to main, Lightdash will update to show the latest version of your dbt project.
You can do this now by setting up a simple GitHub Action using this template and you can read the full instructions on how to do this in our docs here.
At Lightdash, we’re trying to make data development great. Automating tedious tasks and providing a quick way to test will not only increase productivity but also keep your entire team sane.
These features are just the tip of the iceberg when it comes to improving these workflows - going forward we’re taking on an even bigger mission with our new plan to make developing data in Lightdash a joyful experience. Because data teams deserve better!
If you agree (or disagree), we’d love to hear about it. Join the conversation in our Slack Community and let us know what you think!