Why Big Data Is So Complex?


Regardless of how trivial your consultants make it sound, big data is not easy.

One of my clients discovered the complexity of big data when she was tasked to identify the use cases for big data for her firm. Her panic-stricken boss, the Chief Operating Officer of the firm, said to her that he has heard a lot of buzz about big data, that a consulting firm has informed him his competitors are implementing big data, and that he wanted to start implementing the big data solution right away.

It appeared that he knew he wanted big data, but was clueless about where to start.

Based on the directive given to my client, notice the main difference between big data and other technologies. Her boss was not seeking a technology solution for a specific functional area or to achieve a particular business objective.

Well, this is not how we did things in the pre-big data days.

Before the advent of big data, we always had technology applications that benefited specific functional areas. We had CRM for customer service improvement, ERP for overall operational improvement, SRM for supplier relationship improvement … and the list goes on and on. But when it comes to big data, we are not sure which, if any, functional areas can benefit from big data.

What this tells you is one thing….and this is the most important thing to understand for business executives seeking to implement big data: to maximize benefit from big data you may need to create your own business specific use cases.

Wait a second! Did we just use the c-word “create”? Yes we did, and that is the point.

The key idea is that you have to be creative in how you use big data and more than anything else, it is the power of your creativity that will determine what level of benefit you can create for you your firm.

Unlike other standard business applications (e.g. ERP, CRM etc.),when it comes to creating a competitive advantage, for the most part no one will land in your company with predesigned software, predetermined business best practices, and prepackaged configurable modules. When it comes to creating a competitive advantage for your firm and other strategic implementations of big data – you and only you will be the vision architect of your big data.

The above discussion leads us to the first point we need to understand about big data: Big data is not necessarily a prepackaged business application and therefore you would need to “create” the vision of your solution.

And the second point is: Your creativity will determine the business benefit that your firm can receive from big data. In other words, when it comes to big data, your competitive advantage is a function of your creative potential.

Now that I have placed the burden of your big data solution success on you and your advisors, I will make it even harder. Since big data applications are not function specific, we must conclude that they can be applied in all areas of business.

This gives us the third important point about big data: Big data can be used to design an unlimited number of potential applications.

This third law is tied to the first two laws. Like artists having the ability to paint nearly infinite paintings on canvas, theoretically you can design unlimited number of use cases.

The above laws make things a little different than what we are used to.

First, since we have the ability to design infinite number of use cases, and develop applications for all types of functional areas, we must figure out how to identify and prioritize the big data use cases.

Identifying the use cases is complicated. It requires following a structured process to bridge the gap between unlimited creativity and pragmatic business applications.

Unlike other technologies, remember that big data is not automating business processes and it is not providing simple transactional or business analytics. It is building deep insights upon massive sources of information. This means it is processing many different types of information and hence allowing you to create a higher level of abstraction. So your creativity to design use cases would require multidisciplinary knowledge that transcends multiple business areas and knowledge domains. Therefore, when developing use cases, be prepared to include experts from various fields such as linguistics, neurosciences, mathematics, data sciences, psychology, and philosophy … in addition to the traditional business and technology resources.

Our big data solution designs must broaden our consciousness and awareness. If done right, big data solutions can help us identify problems that we didn’t even know exist and can help us identify creative solutions for known and freshly discovered problems. These new levels of insights and self-awareness are a function of the solution design.

Once use cases are identified, they are prioritized on the basis of strategic needs of the firm and other performance metrics such as return on investment. The complexity arises from the fact that infinite variations are possible within the use cases and sometimes adding a unique information dimension can disproportionally skew the cost structure of implementation. Just as costs are subject to design variations, the benefits (e.g. cash flows, risk mitigation, and competitive advantage) are also dependent upon the design. An extra pinch of creativity can result in giving a powerful unique competitive advantage to your firm.

The next dimension of complexity will come from the data management excellence of your firm. It is not enough to have a creative use case or a big data solution footprint. You would need world-class data management practices (data governance, data quality, metadata management, master data management, etc.) to manage and govern your big data. The easiest way to turn your big data project into a nightmare is to not provide the critical support services of data management.

Big data use cases will spread over many areas of business and may require never-seen-before coordination across multiple departments in a company. A hastily developed big data program would quickly run into problems due to political considerations, cross functional dependencies, and territorial confusions. The traditional models of change management may not work in big data due to the supreme creativity factor required to support the programs. Organizations may need to be realigned and new types of change management models may need to be developed to support big data.

Obviously, once the design is finalized, selecting the tech stack that can not only give you a successful implementation but can also grow with your vision would be critical for success. Today, many technologies are available and making that selection would be the fundamental success factor.

As you can tell from the above discussion, when it comes to implementing big data, you would need a host of capabilities, skills, and capacities. Many of these capabilities are not readily available in traditional companies. Many of these skills would need to be cultivated.