There is a lot of hype about big data. Consulting firms are falling over each other to market their big data services. You are being told that your company’s survival will depend upon big data. How much of that is true? Is big data really a revolution?
Now, let us try to address the question: Is Big Data really a revolution?
I base my analysis on some very unique things that have happened in the technology world. Since the advent of the Internet our technology growth has been pretty straightforward. It followed the natural progression of technology moving forward in a predetermined trajectory. Things became smaller, wireless, faster, and more attractive – but the overall information processing frontier stayed the same. We were still processing basic transactional information – information that was derived from and based upon the contractual relationships between entities. From information acquired through the typical business contractual transactions, we tried to understand the common behaviors of people who engaged in those transactions. And our awareness was limited to that – we could model, abstract, and analyze the contractual relationships between humans. Whether they manifested in e-commerce or site developments, financial transactions or buying shares, it was all limited to human contractual interactions.
Obviously, humans are more than the contracts they engage in. And the complexity of human behavior and cognitive processes determines how, when, and what types of contractual transactions they would engage in. Everything we do, we leave traces of information about who we are.
So the biggest difference between small data and big data is that big data now enables us to capture non-contractual information about how humans interact with their environment. This information enables a higher level of abstraction and a deeper insight.
Big data is a revolution because now we can truly model and analyze many different aspects of human behaviors, thoughts, feelings, and actions that we couldn’t do before. We can model and analyze complex human interactions with the environment in which humans operate. And finally we can model and analyze how various parts of our environment act with each other.
The augmentation of human abstraction potential resulting from big data is not like anything we have seen before. The insights provided by big data are far more powerful than what ordinary technology can do. These insights are made available through a variety of tactics including advanced visualizations, pattern recognition, machine learning, and predictive analysis.
In other words, what is transforming is not just our ability to process information but also how we think and cognize information. This cognitive leap is redefining the entire model of information processing. In other words we will be able to think differently about our problems and solutions. See my article Opening the Johari Window with Big Data and Can Big Data help develop our instincts?
So while big data may not be a technological revolution – it is certainly a revolution in how humans think about themselves and their surrounding environment.