Big Theory for Big Data

During the Internet Bubble, sophisticated opinion on Wall Street divided roughly into three camps.

  1. First, breathless enthusiasts embraced the world-changing potential of digital interconnectedness. They developed a new language and new models to justify lofty valuations of entrepreneurial wishes and dreams. These optimists advocated the new rules of asset valuation, often with an ardent sincerity fit for refuting the flat-earth hypothesis.
  2. Second, hard-line traditionalist skeptics dismissed the new rules as a manic delusion, a negligent perversion of the subtle art of  estimating the present value of future cash flows.
  3. Third, the vast majority of agnostic pragmatists refrained from opining on the complex questions buzzing in the Internet Bubble. They espoused the “Greater Fool” theory and focused on the more mundane work of riding the wave as safely and profitably as possible.

In hindsight, the fairest verdict is that all three camps were fundamentally wrong, regardless of how much money they made or lost. Most of the Internet stocks from the late 90s failed miserably, but the Internet really did change the world in ways that make the wildest hyperbole of the 90s sound like polite understatements. Collectively, we misjudged this paradigm shift because we reduced its meaning to its economic value. We were blinded by the profit motive. That’s precisely why most of us failed to profit.

We treated the creation of the Internet merely as an investment opportunity, and we feverishly debated the fair price for the latest dot-com IPO and the value of a click on a company’s website. The truth is that the Internet was not primarily an investment opportunity. It was primarily an innovation every bit as dramatic as its early evangelists claimed. Instead of the obsession with stock-picking, our energy would have been better spent on understanding the meaning and deep implications of this innovation.

We are now making a similar mistake with Big Data, another big trend that we are reducing to simplistic dichotomies. Should we or should we not believe the hype? Can we or can we not make money from this thing? Is Big Data good or bad? The way we frame the narrative shows a poverty of our imagination.

True, data science titillates the modern imagination with grand and strangely credible promises and equally tenable risks. Data analytics can, in theory, stabilize financial systems, streamline the functioning of complex bureaucracies (both public and private), and fuel some of the most urgently needed and economically stimulative innovations. It can also dehumanize us and destroy our privacy. Impossible to deny or undo, Big Data has roused intense interest across the cultural landscape. Hardly any profession, region, demographic category, political or cultural issue stands immune today to the widening sweep of Big Data and the possibilities it has unleashed.

It is premature, to say the least, to deliver a verdict on Big Data, to vilify or to celebrate this phenomenon. It is also premature to formulate a “New Deal” on Big Data that would harnesses the constructive potential of this phenomenon and also constrain the attendant risks. It is also a mistake to frame the discussion of Big Data as clash of hot hype and cold reality.

We see this rush to facile judgment in much of the media coverage of Big Data. Too bad. In our eagerness to act, we are missing an opportunity to think. Big trends need big theories. Let’s stop printing bumper stickers.

/*Leave a Reply*/

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s