A Revolution on the Horizon - Cloud Computing and Big Data

Cloud computing is a big deal, a really big deal. Amazon made $7 billion off it last year. If taking into account Microsoft Azure and Office 365 together, it is speculated Microsoft made nearly as much. Google (or Alphabet Inc...whatever) recently hired the former co-founder of VMWare for her expertise in selling to companies for Google's cloud efforts.

In the last decade, cloud computing has matured. The security argument no longer holds up. Since Amazon launched cloud services a decade ago, they have recruited the talent, resources and learned from experience to become arguably more secure than privately hosted clouds. In 2015 it finally became apparent, everybody from banks to the CIA is now using cloud computing (WIRED).

This is bigger than simply making it easy to get servers. Something else is looming on the horizon, this thing called big data. What is this thing called big data? First, it helps to understand the difference between data and information.

Data is just raw values, a table with columns for SKU (retail product identification code), name, age, address, gender, latitude and longitude. Looking at this table, just scrolling through it and looking at the rows, it is pretty difficult to make much sense of it. This data can be a good place to start deriving information, though.

First group the SKU's into product categories such as cosmetics, groceries, housewares, etc. Next, cross reference the location to demographic characteristics. Finally, associate these demographic characteristics to the product categories to identify trends. This is information, something yielding actionable insight from all the data.

So then, what about this big data thing? Consider this. In 2013, 90% of all data in the world was estimated to have been generated in the two years prior, 2012 and 2013 (SINTEF). We have not slowed down since. The world has accelerated.

In the last few years, the amount of data being collected and stored by various government agencies around the world has come into question. However, the amount of data any government collects on any individual pales in comparison to the digital footprint left in social media and collected through cookies in your web browser. When was the last time you posted to social media from your phone? You shared where you were and what you were doing with the digital world. What have you searched for on Google while logged into your GMail account on Chrome? Google knows what you are interested in. Further, the purchasing patterns collected through loyalty cards (do you save 15 cents at the grocery store using a card on your keychain?) and credit card transactions leave another massive digital trail.

With the holiday season winding down (thankfully!), consider the digital trail you have left. How many times did you share pictures on social media? How many times did you search for something on your computer? How many times did you use your credit card? How many times did you use a loyalty card? This is a lot of data. Now consider you are one of millions. Is it even possible to make sense of all this data? Is it possible to extract any meaningful information out of it? The answer is yes, using big data technologies.

How is big data different than just using a computer to do normal number crunching? Just like in any efficient system, it is about delegation. Big data technologies take a lot of normal computers, just like the one you likely are using to read this article and store all of these almost inconceivably large amounts of data being created across all the computers with a little overlap built in - just in case one computer quits functioning. Next, when the data needs to be processed into information, the instructions for processing the data are sent to all the computers at once. Each computer performs the analyses using the piece of data it is storing and sends only the results back. If you have ever heard the saying, many hands make light work, this is the philosophy of big data. It is all about delegation. Each computer has a little bit of data and does a little bit of the work - simultaneously.

As the data gets bigger, just add more computers. The analysis will not slow down as the data grows since the work on each individual computer never gets more difficult. This is where cloud computing becomes such a big deal. With cloud computing, adding more computers to help support more increasingly bigger data can be done manually in seconds or better yet, completely automated.

This is why it should come as little surprise the CIA is using cloud computing. After a decade of practice, Amazon is doing a pretty good job of security. If the CIA wants to take a look at almost inconceivably large seemingly nonsensical piles of data and trying to derive patterns - attempting to derive actionable information, cloud computing grants on demand access to the scalable computing power necessary for applying big data processing technology to help derive this actionable information.

Now revisit our example of all the data you leave as part of your digital footprint. Your information by itself is not outrageously interesting (sorry to break it to you), but your information, when tied to your home address along with all your neighbors and cross referenced to demographics, this starts to become really interesting. Then this mountain of data starts to make sense, it starts to become information. It becomes possible to understand how far people in a retirement community over the age of 65 are willing to drive to get to a grocery store. It becomes possible to understand how likely people in this same retirement community are to shop online to avoid having to drive to a big box store.

This is the exciting part, being able to start finding answers to questions like this. Cloud computing combined with big data is the marriage of technologies to make this possible. Next year is the year when these two technologies will begin to really work together to become something greater than the sum of the parts. Trust me, I have seen glimpses of the future...and the future looks very interesting.