Shortly after joining Gartner, I noticed subtle magic in the air. Not Magic Quadrants – they were too obvious. It was I told you so repeated by many analysts on many occasions. It seemed almost mystical – how could they know? At some point, I caught myself saying I told you so too — it became natural after seeing and researching so much. For example, I talked about personal analytics, multidisciplinary teams instead of a single data scientist, and about Hadoop being a live archive — these came to fruition and became common place. I told you so. (Mom, I hate when you say it.)
In the big data field research back in 2012, we saw that there was a big data maturity gap. It needed a couple of years to close. I told you so. Glad to report, 2014 was the first year when enterprises became serious about big data: They started asking questions beyond “how do I begin my big data initiative?” or “how do I select a Hadoop distribution?” Hortonworks even went public, the first Hadoop vendor to do so. My colleague Merv Adrian went into great depth on the Hortonworks IPO in his blog post Hortonworks IPO – Why Now?age
The question is — what’s next for big data? First of all, big data will become the new normal sometime between 2016 and 2018. My colleagues Donald Feinberg and Mark Beyer will say (with well-earned pride and a flair of mysticism), I told you so.
Organizations are finally ready for big data in the cloud. In the second half of 2014, my clients started asking about a data warehouse and Hadoop in the cloud. I am an analyst in Gartner for Technical Professionals — 90% of our clients are practitioners who are doing things right now. Therefore, I anticipate many interesting developments around big data in the cloud soon.
In 2015, I expect a plethora of big data applications on top of data platforms (remember, these platforms already demonstrate acceptable maturity). Big data applications will be mostly analytical, and they will be small, in the “app store” style, with few customizations — that makes support and maintenance relatively easy. People would be able to download big data apps they need and use them like Lego blocks to make their own customizations. Big data apps will put a process or a workflow into the spotlight.
If big data apps proliferate, they will need… data. This means a focus on data governance, data preparation and an ability to painlessly load data into big data stores. This also means self-service, or rather SELF-SERVICE. It would be a bigger and bigger subject from the data management and analytics perspectives. And from the process perspective too, I already told you so.
People are impatient. Those who want self-service are demanding increasingly real-time data access, response and gratification. This will lead to in-memory and streaming advances, but I don’t think it will be next: for practitioners, it will be next after next.
The Internet of Things (IoT) is at the peak of inflated expectations of the hype cycle. Organizations collect more data than they can process: for them, it’s still not just about finding the needle, but getting the hay in the stacks. The Internet of Data sustains the IoT. Companies are collecting new data, asking about new external data sources and searching for dark data within. Many new data sources are personal data – privacy and ethics accompany them. This year, I wrote Maverick* Research: Put Your Data in the Bank, Get Dividends, where I foresee intermediaries — personal data banks — representing individuals. A personal data bank will keep deposited data and multiply its wealth through commercialized digital markets. The Internet of Data and personal data banks are not reality yet, but I am sure soon I would be able to say about them I told you so.
