Agile Development Enables DataOps

There are several ways to be agile, but they all share common values and principles. However, the fundamental idea of agile development is to carry out development incrementally in small iterations, deliver often, and get constant feedback to ensure that you are developing the right things. Communication and collaboration are at the core of agile development.

The Natural Path for DataOps

We would argue that most elements of DevOps and DataOps are simply a natural evolution based on agile development. Below, we will go through each of the four core statements of the Agile Manifesto and explain how they are reflected in DataOps.

Individuals and Interactions Over Processes and Tools

In DataOps, tools and processes are one of the main contributors to development. However, a well-functioning team will only work well with effective communication internally and externally. Even the best processes and tools will not help if the product does not deliver what is expected.

This is achieved by clarifying roles and organizing into cross-functional teams that can deliver end-to-end what is needed to create business value. It may sound difficult, but to get started you only need to place everyone in the same room and define goals and tasks together.

Working Solutions Over Documentation

Development should always focus on producing working software over comprehensive documentation. For new solutions, people often talk about MVPs - Minimum Viable Product. This also applies to data products. Then the MVPs can be expanded later based on a business-driven prioritization.

In a well-organized DataOps team, there is also an ‘Ops’ component. This means that the developer within the team is responsible for production support or operations, and should be able to solve many of the potential problems that can arise on their own. They have likely been involved in developing the products, which also makes code improvement, debugging, and bug fixing more efficient.

It will be in the developers’ interest to develop data products that are good enough so that it does not mean constant firefighting. It also means that documentation must be at the right level, or sufficiently good. It can be further developed and iterated on, in step with growing needs for more precision or explanation.

Collaboration Over Contracts

The fundamental idea is that the team should offer the result as a service. Data owners, data developers, data users, and IT infrastructure work together toward a common outcome. This collaboration is essential to ensure that data is handled in an efficient, secure, and high-quality manner. Since we work together, where everyone brings their expertise and background, contracts will not be particularly relevant.

Responding to Change Over Following a Plan

In DataOps practice, responding to change means developing systems and processes that can quickly adapt to new data sources, new analytical tools, changing business requirements, or new regulatory requirements. Through automation, continuous integration and delivery, and robust collaboration, DataOps is designed to handle change efficiently.

Quality Management in DataOps

Lean development and Total Quality Management have over time influenced how we develop software. Here are some key principles that should be applied to data development, operational quality management, and the quality management of data products:

• As in Lean, there should be continuous improvement and elimination of waste in products and processes - create minimum data products and do not make code, data platforms, and solutions more complex than what is necessary right now.
• Optimization should be done for the entire process from design to delivery, rather than just individual parts of the process.
• Optimization decisions should be based on data and measurements, because how can you expect to optimize something without facts and understanding. This is a recurring problem where monitoring, logging, and alerting are not sufficiently in place.