Can Big Data Help Develop Your Instinct?


Humans approach decision-making in two ways: rationally and emotionally. In many cases we apply both rationality and emotionality to formulate decisions.

Many times when we make decisions, we make ourselves believe that our decisions are purely rational when in reality emotions played a major role in our decision-making. At other times we think we are making a completely emotional decision, e.g. loving a person, when there is a completely rational explanation for our decision.

We have all heard stories about new product launches or CEOs pursuing M&A transactions in which decisions preceded analyses?

Who have not seen situations in which numbers were viewed as the true representation of reality and many factors of the reality were ignored or not accounted for?

Reality is often far more complex than our ability to process and model information. And that will be true for a long time even with the advent of big data.

Let us think about the four scenarios in which we can study our responses. Plotted on two axis of rational response (y-axis running from low to high) and emotional responses (x-axis running from low to high) – the area between the two dimensions shows our responses to various situations that we confront in life. This gives us four quadrants:

  1. Automated: Highly rational decisions with extremely low or zero emotion – response purely based upon numbers – e.g. decision to grant credit purely on the basis of credit score, a yes or no response
  2. Highly emotional decisions – response purely based upon emotions – e.g. how we typically perceive falling in love
  3. Low emotional, low rational decisions – response that requires neither rational nor emotional thinking e.g. built-in physiological responses such as breathing
  4. Highly rational and highly emotional responses – responses in which we are dealing with both emotions and rationality e.g. most strategic decisions in business

Let us give some examples of our self-perception of decision-making when we find us in one of the quadrants:

  1. Highly rational decisions: when we use a lot of analysis and data, we tend to think we are making rational decisions. Such decision-making is based upon a model of reality which tries to model the reality based upon some data representative of the reality. This is a preferred method of decision-making for business people – however in many cases we think we have accurately modeled the reality when miss out on several key variables.
  2. Highly emotional decisions: when we base our decisions on our emotions and use very little data, we think our decisions are purely emotional. We believe we are answering to the call of the heart and that such a decision is not based upon some rational data or analysis.
  3. Low emotion, low rational decisions: these decisions are typically built-in responses to pre-analyzed situations. For example, our decision to draw in each breath when we breathe requires neither complex analysis from our conscious brain nor emotional reasoning. It is just there – albeit it is a result of extremely powerful biological decision-making and response.
  4. High Rational and High Emotion state: Obviously a higher state of awareness results when we can apply analytical tools (rationality) to make our decisions or to chart our paths, but we can also understand our emotions and biases. Let us call this super-decision state.

Big data is now moving the first three areas of decision-making into the super-decision state. Super decision state results when:

  1. We have modeled the reality by including more variables than we were able to do in the past
  2. We are processing more information about each variable we have modeled so that we are modeling the reality at a deeper level
  3. We are processing information both about the state of the reality and also about our sense or perception of the reality – it helps us understand our biases and bridge the gap between actual reality and our state of awareness of the reality.
  4. Finally, it is the zone in which we can study the interactions of interdependent actors that are adapting to a dynamically developing system.

We do that by capturing and processing different types of information, by creating new types of relationships between information, and by deploying constructs that can go beyond simple number based analysis.