Data Platform | Can it be built in 6 weeks?

Building a new data platform from scratch requires a great deal, and it is important to think through both the process and what to deliver first. We present 5 steps you should follow to deliver value within just a few weeks.

5 steps to deliver value in a few weeks

The short answer to whether it is possible to build a data platform in just 6 weeks is “it depends”. But to give a better answer, we take as our starting point what we mean by a cloud-based data platform, which you can read more about here. Generally, we refer to all the various components and resources in the cloud needed to collect, store, process and make data available for analytical purposes.

Both designing and developing a complete end-to-end data platform in just 6 weeks is probably unrealistic. But it is possible to get started. And it is possible to deliver value.

To do this, you can proceed as follows:

1. Define a narrow scope (MVP) for the first use case to be supported. Typically only 1 source, with simple data transformation. The platform capabilities you build first are typically the ability to ingest, store and process data. Monitoring, full CI/CD and the like can come later.
2. Work as an integrated team – and be in the same room for most of the time. Business owners, data engineers, report developers, data scientists and data platform engineers must work together.
3. Ensure you deliver business value – then it is easier to continue working on platform capabilities later. Also plan for activities that deliver value all the way to the end users – and do not stop at the technical deliverables.
4. Include experienced data platform developers – and a competent scrum master and driver who brings everyone along on the journey. Make sure to clarify dependencies early. 6 weeks go quickly if it takes 4 weeks to get the right access permissions.
5. Create a plan to further develop the data platform capabilities – in step with the use cases. There is little point working hard for 6 weeks if the work stops in week 7…

If we assume that you get these 5 points in place, together with a clear understanding of the needs the data platform should address, six weeks can in fact be enough time to develop a foundational version for continued further development later.

More to come – you need to manage expectations

After getting the basic capabilities in place and having delivered value, it is important to remember that you are never finished. A fully-fledged data platform for a larger organisation requires a number of supporting functions beyond the core capabilities, such as a data catalogue, monitoring, CI/CD, external data sharing, access management and more. In most cases, it is these functions that serve as the glue holding everything together.

In other words, you must pay attention to expectation management. The work is not finished. You must agree that resources need to be allocated going forward as well, but that the plan and scope are clarified gradually – and the first 6 weeks set the course.

Two approaches: 6 weeks, or the peace to do the right things

We have several approaches to building a data platform. There are at least two variants:

The variant where we move at top speed to get something up and running within 6 weeks is for situations where we need to build competence. Where data engineers and other key stakeholders must be part of the journey from the beginning. Where it is not so clear to them what the new approach entails. This is perhaps most suitable for slightly less mature organisations.

Ideally, I believe the best working approach is where we do not have to work intensively for 6 weeks, but the team instead gets the space to work on what needs to be done over time. This applies primarily when you have a competent data platform team comprising a mix of internal and external resources from day one, and where there is agreement on the capabilities that need to be in place first.

What do you think?