In recent decades we have witnessed breathtaking advances in technology. Information technology (IT) is growing exponentially, writes Professor Andy Koronios, and it’s delivering huge benefits to the way we work, interact, play and look after each other.

The widespread adoption of mobile phones and other ‘smart’ devices, autonomous systems, and the billions of sensors and sensor networks, along with growing social media connections, are going to further dazzle us with what is possible.

We live in a truly intelligent, instrumented and interconnected world and our IT-mediated lives are leaving a huge vapour trail of data. It is this virtual trail of physical reality that is the best definition of Big Data.

Consider, for example, how much you are paying for a flight next time you travel. If you ask the passengers sitting next to you how much they paid, it is likely none of you paid the same fare for the same flight in very similar seats. Why is this? More importantly what is the optimum time to purchase a ticket so that you get the best price? You might think that it pays to buy the tickets early, but that’s not always the case.

A traditional approach to solving this problem might involve analysing some data to create some rules to program into a computer application, which will then predict the best time to purchase the tickets.

But what if you had the data of all the past flights ever made and the prices of all the fares that every single passenger paid? You could then get the computer to analyse and find the correlations that would give you the best fare. This would have been inconceivable only a few years ago; today it is reality. This is Big Data.

Technology allows us to access and fuse together multiple data sets to solve large problems. Oren Etzioni, an American-Israeli Professor of Computer Science, used Big Data to do just that. He started Farecast, a company that provides intelligent airfare advice by predicting when a passenger should buy a ticket to get the lowest fare. It does this by analysing 175 billion data points of previous airfare data and can predict whether a fare will go up or down with more than 70 per cent accuracy. Farecast was acquired by Microsoft recently for $US115 million and has been expanded to hotel rooms and other services.

So is Big Data a big deal?  If you don’t think so then you mustn’t have yet witnessed its transformational power. Big Data innovation will affect every industry and activity and will fundamentally change some of them, creating new economic value in the process.

There is a plethora of examples where our virtual trail of data has led to innovation with transformational capability.

Take Google Maps for example – it has fused mapping with GPS, mobile tower and WiFi data to give us a product with powerful navigational capabilities. The latest Big Data extension to Google Maps is traffic prediction, which allows you to view in real time areas where traffic congestion occurs.

In tandem with the emergence of Big Data, a new breed of professional has now emerged: the data scientist.

Some reports suggests the age of Big Data will generate thousands of jobs, and data scientists are already one of the most sought-after professionals in the ICT sector.

So what is a data scientist?

The skills and qualities that make a good data scientist are difficult to find in one individual. Some have described a data scientist as ‘part analyst and part artist’. Data scientists must have a solid foundation in statistics and computer science but they also need to understand business and have the communication skills to discuss data insights and convert them into business value propositions for an organisation.

UniSA has responded to industry demand for developing data scientists and was one of the first universities in Australia to launch a Master of Data Science degree, in partnership with leading analytics company SAS.

Big Data isn’t set to transform our lives – it already has. We now need to harness its power to drive innovation in all industries, from marketing to health, from government and to defence.   

Professor Andy Koronios is the Head of the School of Computer and Information Science in the Division of Information Technology, Engineering and the Environment at UniSA.

Find out more at the Data Science at UniSA website.