Data analytics journey: the process that takes data projects from abstract ideation to concrete success

There’s a wrong way and a right way for a company to solve a problem using data. Companies that jump into building a data solution without defining the “why” behind it likely will be disappointed in the results. An approach the moves from ideation to prototype to execution will more likely result in progressive and satisfying solutions.

The first article in our series highlighted that data maturity and organizational transformation go hand in hand. An organization that gets better at harnessing data will see transformation in its people, processes and overall business results.

The second article focused on the different roles and responsibilities an organization has to fill as it becomes better at understanding and using its data.

This article focuses on the process for a successful project and it applies to any company regardless of where they sit in the data maturity model. This is a simplified Agile model for a data analytics project; adopting this process within a company is the fastest way for a company to advance across the maturity model. The process will effectively move from an abstract process that helps a company understand the core problems limiting the business to a concrete project aimed at solving that problem.

An effective data analytics project has three parts: ideation, prototype and execution. These steps are similar to the roles and responsibilities discussed in the previous article. The ideation process includes the executive sponsor, program manager and project manager; in a $1 billion organization about 10 people would be involved in this step. The prototype stage is driven by the program and project managers and focuses on a specific group within a company, such as a singular business unit or functional area. The execution stage implements the work done in the first two stages throughout the entire organization, and features the work of the architect, business analyst, data analyst, data engineer and data scientist.

Ideation

Before a company jumps to building a solution, it has to identify the actual opportunity. The ideation process is what a company, often working with an outside advisor, will use to identify pain points and consider possible solutions. A client company often will say it needs access to all its data as part of the search for a solution. The ideation process will help a company identify its main pain points and opportunities, and then the necessary related data.

The ideation team uses focus groups to identify key company goals and the pain points that get in the way of these goals. At this stage, empathy is key – asking questions and intently listening to answers. This is how pain points surface. These could include problems with accounts payable, manufacturing processes, new product development or supply chain issues. If the company wants to enter a new market, what do they need to measure about that market? If the company is trying to change the behavior of its sales people, how do they measure current sales practices and desired future sales techniques?

Next, the team defines the personas impacted by the pain point. Personas help the team visualize an actual person who will benefit from the solution – a new product developer, a sales manager, a customer or a key supplier. 

After identifying the pain points and personas, the team can ideate where the best opportunities are, the data elements necessary to drive those opportunities, the relative value of pursuing one opportunity over another and possible solutions. For example, while accurately measuring accounts payable outstanding is important, it’s less valuable to the organization than understanding which are its most profitable products.

Solutions will vary depending on the problem and where the company sits on the data maturity curve. The solution may be as simple as identifying and collecting data in a more efficient way. It might be developing a data dashboard or a data warehouse. At this brainstorming stage, there may be more than one possible solution to a problem.

When considering solutions, the team also has to consider the necessary resources for successful implementation as well as undertake financial forecasting. Each solution addressing a pain point requires a unique data source to support them.

Prototype

In the prototype stage, the team takes the possible solutions identified in the ideation phase and goes through iterations of building a prototype, measuring whether it is doing what the team expects, and learning from that measurement and adapting the prototype. In practice, the team is building a data analytics tool, getting feedback from the personas most affected by the analytic, and adjusting the tool as necessary.

Execution

The next step – execution – is applied across the entire organization. Teams that execute well will have a known “velocity,” i.e., the number of backlog projects it can complete over a given period of time. In this execution stage, the business analyst creates and prioritizes the project backlog – an ordered list of work with short descriptions of a product’s features and fixes. The developers – the architect, data scientists and data engineers – are the ones who execute the backlog. They do this in part by developing story points – units of measure to estimate how long it will take to fully implement a product backlog item. The story points enable a team to define the velocity.

A key part of this step is the cadence that develops between the business analyst and the developers, in which a backlog is continuously created and handed off systematically to the developers, and then back to the analyst. Teams that don't have a consistent cadence for getting feedback from users run the risk of building ineffective solutions.

Part of the work that happens here is accounting for the “technical debt” that accrues because of the speed at which new data analytics tools are developed. Technical debt is basically the extra rework that is necessary as imperfect new software is rolled out. Project teams should allot 10 to 20% of its development effort for reengineering what has already been built.

Companies know that they don't have the factual evidence – the data – to make decisions at a speed that successfully supports and grows the business. They may have had some success at guessing what the right solution is in the absence of factual evidence. Guessing, however, is not a competitive advantage.

Successful continuous integration and deployment happens when a company can easily roll out new data analytics tools over time. Agile is the framework for a data analytics project. Technical debt is a key element that has to be accounted for in a project. What the company gets out of the process is a continuous flow of code and successful projects that it can leverage throughout the organization.