According to research company IDC, data creation is occurring at record rate. In 2010 the world generated over 1ZB of data. By 2014 we will generate 7ZB a year. Much of this data explosion is the result of social media, sensors, smartphones and tablet computers.

Big data involves the extraction of value from a wide variety of large volumes of data. According to a recent McKinsey Global Institute research study, the value of big data is $300 billion annual value to the U.S. health care industry alone. This may seem somewhat ambitious, but it highlights the potential of big data analytics. Mining data from unstructured sources such as twitter feeds and internet blogs, and mashing it up with structured data can produce critical market intelligence.

One of the many new sources of data growth is the healthcare industry. The rapidly evolving healthcare environment is transitioning to electronic medical records and images, introducing sensor networks, utilising health related mobile applications, exploiting RFID tagging and consuming social media. The unstructured data available from these sources (accompanied by structured data within healthcare organisations) can provide enormous insights into fields such as public health monitoring, long-term epidemiological research programs and health trends. Furthermore, with the evolution of mobile telephony via smartphones and tablets, additional data to harvest includes geographic location, text messages, browsing history and motion (via accelerometers). Researchers have already started creating models, using data from Twitter, that enables you to see the spread of infectious diseases, such as flu, throughout a real-life population observed through online social media (http://tinyurl.com/dx8dldd).

Unfortunately, traditional business intelligence (BI) tools may struggle to keep pace with big data. Many BI tools simply are not designed to sift through this much data, or identify meaningful (versus un-meaningful) data. As the amount of data continues to increase, data discovery capabilities will become increasingly important, and existing BI tools will need to evolve to keep up.

For healthcare providers, we need to be aware of not only the potential benefits of big data, but also the challenges that big data provide. According to IDC, these challenges include:

There is enormous promise in big data analytics, where it could yield a competitive advantage and real consumer health benefits for those willing to step up to the challenge.