According to IBM, the world creates 2.5 quintillion bytes of data every day. Known as “Big Data”, this burgeoning Big Bang draws its energy from countless interactions in digital media worldwide – websites, blogs, cell phones, corporate and government databases, text, images, video, transaction data, etc. The digital universe is expanding at an astounding, if not a disturbing, pace, and we are certainly not seeing commensurate increases in our ability to store the mounting data, to make sense of it, or to assess the risks and opportunities it may pose. Not a single area of life will remain untouched by this juggernaut.
I first started thinking about the increasing volume, velocity and variety of data creation and duplication before the dot-com crash of 2000. At that time, the challenge of information overload and the promise of improved storage and analysis was still a nascent story. Today the story is undeniably relevant to all of us in some way, regardless of profession, age, gender, location or personal priorities.
Consider these recent tidbits highlighting the significance of the Big Data trend:
- 90 percent of the digital data in the world today only came into existence in the last two years. (Source: IBM)
- Global mobile data traffic in 2011 was more than eight times greater than the total global Internet traffic in 2000. The annual growth of mobile data traffic from subscribers in emerging markets is expected to exceed 100% through 2015. (Source: Cisco)
- In 2005, the digital universe weighed in at 130 Exabytes, growing to more than 1200 Exabytes in 2010, projected to reach more than 7900 exabytes in 2015. Over the next decade, the number of “files” in the digital universe will increase by 75x. (Source: IDC)
- The amount of information individuals create themselves — writing documents, taking pictures, downloading music, etc. — is far less than the amount of information being created about them in the digital universe.
- Walmart handles more than 1 million customer transactions every hour, which is imported into databases estimated to contain more than 2.5 petabytes of data – the equivalent of 167 times the information contained in all the books in the US Library of Congress.
Indeed, the trend is too big to ignore, and corporations seem to understand that, as they scramble to develop increasingly sophisticated analytic tools to help them interpret the data and use it to explain or predict consumer behavior and economic trends. This New York Times blog post provides a nice update on these data-driven innovations.
But data on Google search trends suggests that the mainstream tech user is also beginning to grapple with the implications of Big Data.
The interest is understandable. Just as Big Data can help businesses make better decisions, it can serve the same purpose for individuals. Given the increasing data-dependency of many of our corporate and individual decisions, we can probably benefit from greater fluency with the core tools and concepts of data analysis.