DataOps | Making the Case for a Central Data Platform Team

With new architecture comes the opportunity to organize differently. We believe this opportunity should be seized, and that you should especially consider establishing a dedicated data platform team

In the old days, we typically operated with a single team that handled operations of the data warehouse and tools (on-premise), developed ETL/ELT and built standard reports for the organization. We had a unified technical community, but faced challenges as the capacity for developing new solutions declined dramatically over time, since we accumulated ever more to maintain.

Platform capabilities were relatively stable and only changed with tool upgrades. These upgrades consumed most of the team’s capacity during the periods they took place.

Cloud-based architecture brings new joys – and sorrows

In the data lakehouse world in the cloud, we have the opportunity to organize differently. We continuously gain new capabilities and functionality that we can adopt. We no longer need DBAs to manage the databases, but have a great need for platform developers and architects who are able to assemble components that support the needs. And the needs are now broader: we have gone from reporting only, to self-service, advanced analytics and real-time streaming. This creates many opportunities, but is more demanding to navigate.

With DevOps/DataOps and agile thinking, we approach deliveries to users differently. We work together in teams with users, with deliveries that evolve in step with the needs.

Mesh or not, we have increasingly decentralized development. Transformations and logic are developed in close collaboration with users who have domain knowledge, and in many cases by the users themselves. With DevOps, we do not have handovers from development to operations – those who develop are also responsible for operations – because operations is really just continued development in smaller steps.

What does this mean for data platform team organization?

We make the case for having a central data platform team, with clear responsibility for the operation and continued development of the data platform service, which is used to deliver various forms of data products by many different teams and environments.

We find it demanding for one person to master – and have time for – being both an ELT developer and a data platform developer, which is a combination we see frequently. It becomes more enjoyable and leads to better solutions when we specialize. Data platform developers contribute to better functionality and increased stability, performance, security and cost-effectiveness. Additionally, specialization increases the likelihood that new capabilities are developed correctly. That makes good sense.

The data platform team should not be responsible for developing MLOps, datasets and reports, forecasting models, etc.

Everyone of course needs good ELT developers and data architects, who

• Enjoy developing pipelines in a cloud-based infrastructure
• Like setting up transformations with SQL
• Thrive with data modeling
• Have an interest in data streaming and related solutions
• Find it exciting to develop data products

Everyone also needs good data platform developers and architects, who

• Find it exciting to build solid and scalable data platforms
• Thrive with setting up CI/CD pipelines and building data platform as code
• Enjoy working with data architecture
• Like building robust and automated pipelines for ingesting and processing data
• Are comfortable with agile methodology and deliveries with continuous capability development

Here is how we think it should roughly work:

Are you a data platform or ELT developer who is curious about the professional community Glitni is building? Feel free to read more about how we maintain a primary focus on data – every single day.