This is part three of a five part series on disruptive technology. You can read Part 1 on IoT here and Part 2 on AI here. Disruptive technology is technology that pushes the boundaries of an industry, dramatically changing how we live or how things are made or done. The term disruptive innovation comes from the book The Innovator’s Dilemma by Clayton Christensen (1997). Many, many people have written tons of examples that are old-school: hard-drives/floppy disks, monitors/flatscreens, and so on. This series is about new technologies (at least new in 2017/2018) that are disrupting the world today.

Big Data

What exactly is big data?

Big data is data.  What is data?  Data, at least in modern times, is digital representations of information.

Okay, but what do people mean when they say big data?

Actually, they mean a lot of things.  Here’s a breakdown of what they might mean, and helps to define “Big Data” today.

1. Cheap storage – because it is cheap to make and use digital storage that is actually quite fast to access, new possibilities for what we measure and store are possible.
2. Fast and easy access – cloud computing has made it possible to quickly search, sort, and analyze data… easily, from anywhere.  Without buying your own server farms or super-computers.
3. Tracking everything – since cheap storage is fast and easy it is therefor possible to track and collect data on more and more things. This is further enhanced by IoT and the massive adoption of the internet which makes it easy to instrument data collection tools.
   1. For example: Google Analytics can easily track lots and lots of things about people using and uses of your websites or apps, all at zero cost to you.  Free Big Data!
4. Statistical Analysis Tools – since it is fast and easy to access lots and lots of data.. and we are saving lots of things… new algorithms such as predictive algorithms in statistics make it possible to “find nuggets of insight” by analyzing the data.  This analysis can be cheap because of cloud computing.
   1. Predictive Analytics – this is analytics that seek to find potentially causal relationships between desired outcomes (likes sales) and customer actions (like looking at testimonials).  It can analyze tons of potential customer actions to see which ones most commonly lead to sales.
   2. Cohort Analytics – this is analytics that create groups with like features (such as age-groups) and analyze what might be different about their usage or their predictive analytics for that group.
   3. SO MANY MORE!  I cannot list all of the great statistical things you can now to with Big Data…. it can also power and fuel AI training engines!

So Why Big Data?

Businesses are trying to leverage “Big Data” to do the following things (and more)…

1. Better Marketing leading to Increased Sales – by learning more about their customer, by doing better and more customized marketing, by doing better at attracting the right customers, etc.
2. New products – Big Data lets companies make new things possible, like counting calories, comparing your walking distances and habits with friends (Fitbit), knowing what kind of music you like (spotify/pandora), and so many more!  Data is cheap, so you can use it to make amazing things that were never before possible.
3. Lower Costs – Digitization can lower costs, and now since Big Data is cheap, anything can be digitized cheaply and usually pretty easily.
4. Here’s a few more:  Track Things, Monitor Things (uptime), Make Things Safer, Make Things more Robust, Predict Failures, Predict Outcomes, Simulation, and oh so much  more.

Here is a pretty good TedX about Big Data if you want to learn more:  https://www.youtube.com/watch?v=qXyzDd2heK8