Data Platform | Get Control of Cloud Costs!

Data processing costs can easily spiral out of control in the cloud. We have some simple tips to combat high costs - you wouldn't want to give away money when it's easy to do something about it, would you?

Simple steps to keep cloud costs down

In one of our projects, we have focused on and managed to reduce costs recently. This comes from some simple changes:

1. Ensure transparency: make sure that both developers and product owners see the costs that data flows and models create. In other words: tag data flows and processing logic, and create cost reports and predictions based on the tags.

2. Choose what to focus on: do not try to optimise all data flows and models at the same time. Identify and take action on the 3-5 most cost-intensive elements at any given time.

3. Keep it simple: Small changes are often enough. For example, by switching from full loads to incremental loads, adjusting job scheduling to actual needs, or reducing the capacity of a resource group.

4. Include performance and processing cost in testing: ensure that a performance focus is part of the development practice, and included in pull request templates and peer reviews. Can queries and logic be set up more efficiently?

5. Ensure a modern storage architecture: use ELT, and make sure that data is moved and duplicated as little as possible to avoid unnecessary processing and updates. Some possible measures include logical rather than physical storage, and only one environment where source data is loaded which is then accessible to development and test environments in addition to the production environment.

Use the technologies’ built-in cost control features

Many cloud technologies have good built-in functionality for controlling costs. You can find information about this on their documentation pages, or with a simple search on YouTube. This video, for example, provides a good introduction to how you control costs in Google BigQuery:

Don’t waste your money — use it to build great data products instead!