This article will speak about democratizing data-driven decisions with self-service tools. We are talking about democratization because we are going at a swift pace, and technology changes. Things are changing so often that we have to meet those changes and make respective changes from the technology point of view.
In a traditional approach, whenever we get into any project, the business team sends us the request for whatever information you’re looking for. They send their requirements to the BA or executors. They send this request to the IT department. The IT department develops a solution, processes it, shares it with the executors, and then presents and explains it to the business. This approach is good and works well in typical day-to-day life, in product development cycles, and even in various departments.
So why do we want to change this? First of all, we face a few challenges while following the standard cycle. Business requests change often. The data that we are processing also changes quite often. The workload of the IT department increases while taking this requirement and developing something like that. It could also be time-consuming.
To avoid that, we have another example. When you are looking for any information, as a layman, you type a request on the browser and have the results on the screen. It is easy to access this information because of the layout and the technology behind it; the algorithm gives you what you are looking for. This is something we have made or modified. When you’re on the business side, you can access a Big Data platform and multiple tools on the platform. But, this can be difficult from the business perspective. Data is a new fuel, and we all have this necessity to understand and learn technology as much as possible, even from the business side. These points help you achieve this from the Big Data platform.
Let us consider a recently captured business case.
These are the actual numbers. A team spends almost two months working on or delivering the project; you receive information, they process the things, and then there is a cycle of UAT and production. Once we move to production, we have many other challenges which add to the time frame. This timeframe can be reduced to 45 to 50% by democratizing data and using a platform. To give more details, I want to share one small analogy.
For example, I will compare an application with a restaurant and a platform with a cooking studio. When you go to the restaurant, you get a menu card. You can select any dish from the menu and get it served. When you relate it to an application, you select the software you want to use, and you get your project delivered. In the cooking studio, you get a price, or you get access to the backstage, you get all the ingredients, and you get all the access to different details in the cooking studio. So here, you can mix and match different ingredients and create whatever you wish to. There are chefs to help you understand the ingredients and other details. This is similar to a platform where you can access existing tools and build your own.
This framework is from the business perspective; when we adopt the data-driven culture, we follow four As – Ask, Acquire, Analyze, Act, and then deliver an outcome.
When you are a business, and you make a request, we, the development team, define the problem statement, check the feasibility, design the solution, analyze, create different models, act on it, i.e monitor and get the outcome (deploy). We check to see if the requirements are met. If requirements are not met or business changes requirements, the entire process has to be followed again, and this will take a longer time for delivery.
If different product teams develop different solutions, we can create a central repository. This repository is accessible to all teams, which makes code sharing easier, and we do not have to develop from scratch every time.
How does data flow in a data platform?
Data comes from various sources like files, streaming files, and different databases. On a central platform, we have multiple layers, data storage, computation, analysis, and governance. Governance plays a vital role on a central platform.
When we process data, consumers of this data could be data stewards, data analysts, scientists, and engineers. Their methods of accessing the platform will be similar. We use a self-service portal, where all the tools in the platform can be searched and accessed. A “Centre of Excellence” team supports this platform. The key is to have a Centre of Excellence to support the platform, with suitable training material to help them guide wherever and whenever they face any challenge. When we have 50 to 80 tools in the platform, microservices and APIs play a vital role in passing data from one application to another application or for any other processing. Throughout the process, microservices play a vital role in achieving your outcome.
Tech innovation that propels data localization includes three building blocks,
- Tools and Infrastructure
- Governance and Strategy
- Data and Insights
Center of excellence, data visualization software, data cloud storage, BI tool, and e-learning workshops play a crucial role.
What has changed with data democratization?
Cultural change and tools adoption rate has led to faster data access. Data discovery provides a broader view of the metadata and avoids silos.
It needs a lot of effort to change and work on something new. Not many people are used to it, but this is a constant change. And we have to adopt that. When we democratize data, we train users and give them ideas on that. And this is how cultural change helps to implement this. We also avoid silos here because of the central repository that we follow. We get access to everything, and there is no dependency on other people.
Of course, nothing comes very easy, even though we find it very simple to implement data democratization.
Business users getting full access to the Big Data platform may misinterpret or not understand data well. They develop something that does not meet business needs, and it is not delivering what you’re looking for. So to overcome these challenges, we need to emphasize education and support to the business side.
Security is also a concern as data is accessible to everybody. So, the tools that we choose need to address these issues. We need to add enforcement points on every tool whenever data flows from one application to another. Tokenization is also essential. So these two things help to mitigate these challenges.
The key takeaway is to make data democratization successful; cultural change is important to adapt to a self-service environment.
