What Is The Key Concept Behind Big Data


Even though they can feel that something big is in the air, non-technical business leaders find technical definitions of big data confusing. They can see the excitement building and they hear constant murmur about the potential of big data, and that is why many executives would like to quickly understand how to capitalize on this great opportunity. In this article I will provide them a different view of big data – a view they can use to understand big data quickly, and more importantly, they can use this understanding to construct powerful solutions for their businesses. I will explain the key concept behind big data.

The evolutionary leap

Let me begin with an analogy from the theory of evolution. Some evolutionary scientists now claim that bipedalism (walking on two feet) had something to do with the development of the human mind. These scientists claim that a bipedal animal is a more efficient energy user and a more proficient information processor. Walking straight, bipedal animal can get a broad 360 degree view of its surrounding, it can observe the land and the skies, and it can be far more aware of its vicinity.

The point is that the ability of an animal to access and process greater information is directly tied to its ability to acquire an evolutionary advantage. Let us call this Reach. Reach is the ability of an entity to expand its self-awareness by accessing and processing greater information.

Next, let us take a moment to understand the type of information processing that our evolutionary ancestors did. As we learn more about the human civilization we realize that we are gifted with the unique ability to conceptualize and develop ideas. While our brethren in the animal kingdom process basic functional information of built in physiological processes (e.g. hunger, feed, mate, excrete etc.), we can develop and communicate creative thoughts. In other words, while the entire animal kingdom can process basic structured information, what makes us uniquely human is our ability to process and use information to create powerful ideas and develop concepts. Let us call this ability Use of information.

Use is the ability of an entity to process and abstract information to develop complex thoughts, ideas, insights, and concepts.

Needless to say that by combining Reach and Use, humans were able to acquire an unparalleled competitive advantage in the animal kingdom.

In fact, one can envision an imaginary space resulting from the product (multiplication) of reach and use of information. We call it the space of competitive advantage.


Competitive Advantage = Reach X Use

Now let us combine the ideas of reach, use, and competitive advantage to develop a definition of big data. Hence big data is defined as: Big data is the expansion of an entity’s information reach and information use to acquire a unique competitive advantage.

Dissecting the three parts of the definition

First, accessing and processing greater information means to be able to capture a lot more information about what impacts us. Like the analogy used, our evolutionary ancestors were trapped in the most rudimentary information processing (response to basic survival processes e.g. hunger, mating, thirst, etc.). Lost in the basic transactional information processing, we were acting like animals that process only the most basic functional information. Our analytical capabilities and response potential were limited to basic functional responses. Our abilities to comprehend the problem domains were limited to most obvious problems. Our capacities to formulate solutions were limited to linear and rudimentary solutions.

When we expand our information processing horizons, we move beyond the basic rudimentary insights about our environment. This means we need to be able to process complex information that includes the structured (i.e. the basic functional or transactional information) as well as unstructured and semi-structured information.

When we designed our information technology systems, we only focused on capturing and systematically analyzing the most elementary information about business processes and transactions. We ignored a lot of information that was not structured e.g. we had no automated means to systematically analyze the information talked, written, or shared about us in the form of articles, customer comments, shareholder sentiments, analyst analysis, employee emotions, and competitors’ description of who we are or what we do. All of that information is unstructured information and it escapes the net designed by our IT departments to capture and process information.

Second, being able to process greater information is not enough. What we need is a new level of abstraction that can give us new insights, a higher level of awareness, and a deeper consciousness. Thus we need advanced tools to enhance self-awareness and augment competitive consciousness. These tools come from advances in areas such as virtualization, predictive modeling, machine learning, and pattern recognition.

Third, our enhanced ability to process and analyze information and develop complex ideas must result in giving us a competitive advantage that can be on the same scale as the advantage gained by humans in the animal kingdom.

In summary, big data is increasing both our efficiency and effectiveness in capturing and using information to create new levels of competitive advantage.