Ever since the term Big Data was coined in 2008 by Haseeb Budhani, it has become an absolute must have in the domain of emergent technology – something you must know about as an organisation if you hope to compete in the new era of the connected Internet of Things. The reality however is that the true value of Big Data has been in the conversations and paradigm shift it has created around the use and value of data.
Strictly speaking, Big Data relates to a set of tools and technologies that allow a distributed array of computers to process raw information generated from machine code, server logs or smart sensors at a much higher velocity than traditional data management processes allow. These examples of raw information have become known as unstructured data that is data that does not have a uniform make up and has not been processed into a constrained relational database management system where certain field types, lengths and conventions would need to be obeyed.
Technologies such as Hadoop maximise the efficiency that multiple computers can cope with distributing and prioritising tasks. By using efficient task distribution across an array of computers, tasks that previously would have taken weeks, if not months to complete can now be completed in a matter of hours, if not minutes.
Big Data enables organisations such as NASA to register millions of tiny sensors on each of their launches into space, capturing billions if not trillions of information points in the space of minutes and then be able to process those information points into aggregated data points that are far more useful for analysis. Big Data by itself, of itself, will not generate analytical insights or create valuable information, Big Data can best be thought of as a tool that takes information that could not previously be analysed due to the scope and unstructured nature of the data, and make it accessible for a predictive model or insight generation tool in a way that can then be analysed.
So, how many businesses have truly trillions of rows of information being generated in just minutes that they need to process and then analyse….? In many cases, enterprise organisations seek to adopt Big Data technology believing that it will be the silver bullet they need to be more competitive, however very few of these organisations have a business problem that can truly be solved with pure Big Data technology, which brings us back to the question – what is the true value that Big Data has provided for the majority of organisations?
Big Data has created new conversations that have defined the importance of information, the realisation that data is truly an asset for businesses and the realisation that externally available data through things such as social media, open government datasets and other data providers is an absolute key to unlocking the predictive potential of data within your organisation. With many case studies, articles and success stories showcasing how organisations like Samsung were able to half their marketing budget through the use of data driven decision making while tripling the level of engagement they received from marketing, these conversations now happen at an executive level, empowering data management and modelling professionals with the mandate to seek out and explore new ways of generating insights that can be used for competitive advantage within an industry. In the majority of instances, connecting to social media services such as Twitter or Facebook to gain sentiment analysis of how your brand may be perceived in market do not require expensive investments in Big Data technology and can be handled by many traditional data management and relational database systems already in use by organisations.
However your data strategy must prioritise the importance of using information and creating actionable insights, rather than just processing and storing the information in vastly expensive data warehouses that may or may not actually provide the business with anything that enables them to make better decisions. Building the type of data strategy that creates a data driven culture within organisations can no longer be left solely to technology professionals. Likewise, the skills to analyse and interpret data points is becoming equally as important as knowing how to use tools like Microsoft Office.
The hype and conversations generated by Big Data have played no small part in the shift away from a technology driven approach to data management towards the business driven approach to the way data is consumed. The best approach to gaining value out of your various data sets and new sources of information will depend on the type of business problem that needs to be solved. Fast tracking the development and deployment of a tool before understanding what benefit you're attempting to achieve only puts you on the fast track to failure.