Big data and analytics came to the forefront in May 2011 with the publication of a seminal paper by the McKinsey Global Institute titled "Big data: the next frontier for innovation, competition, and productivity". The Big Data and analytics boom peaked in June 2016, according to Google's Trend Analytics (which increases interest in keyword searches). And cloud computing has continued to receive a high level of attention as more organizations continue to implement cloud technologies to increase business agility, operational elasticity, improved performance, and greater efficiency.
Digital transformation needs to work at the organizational level and will become a permanent way of operating.
People may wonder what will become of big data and analytics after they reach their peak of development. As long as published customer surveys, vendor interests, analyst reports, revenue sources, and other information are of value, then organizations will adopt big data and analytics to access them. A 2016 survey conducted by research firm Gartner, Inc. reported that organizations' investments in big data and analytics have been growing over the past five years, but interest in their future investments seems to have declined. This could be due to a kind of lull in getting tangible benefits from these investments. And another Gartner survey reports that only about 12 percent of big data projects have achieved measurable results. However, technologies such as social media, the Internet of Things (IoT), smartphones, mobile devices, gaming gear, wearables, sensors, drones, remote monitors, precision medicine, precision agriculture, smart cities, smart buildings, self-driving cars, remotely controlled vehicles, and more, will generate an enormous amount of data that needs to be collected, aggregated, and analyzed to make useful and valuable decisions.
And it is impossible to analyze data manually using traditional methods and systems. The potential value from big data and analytics is in the billions of dollars per year. This is considered a conservative estimate. That's because a survey conducted by McKinsey & Company in 2011 reported only a small fraction of the potential value of big data. Only location-based data has a high adoption and value capture rate of 50-60%, followed by the US retail sector at 30-40%, manufacturing at 20-30%, the US healthcare sector at 10-20%, and the EU public **** sector at 10-20%. As a result, interest and investment in big data and analytics will increase in almost all industries to capture the hidden value in big data. It is expected that there will be continued interest in big data in the cloud by organizations in the coming years.
Data security
As more and more data is collected, aggregated, analyzed, and used to make decisions that affect people's lives, data security has become a top concern. Data governance entails dealing with the peaks of data collected from different sources and the central stage of managing the risks involved in these data elements. U.S. federal, state, municipal, and local government agencies and other nonprofit public ****service organizations are required to meet stringent Confidentiality, Integrity, and Availability (CIA) rules and also provide good governance, meet compliance requirements, and manage risk (GCR).
A common misconception is that organizations need large amounts of structured and unstructured data collected from different sources, including external sources (which require validation and risk assessment) to start analytics. Organizations don't need a lot of data to start an analytics program. They can start with the "gold standard data" that is already available and consider the possibility of using it alone or in combination with other internal data sets to address business questions as proof of concept for purchase from decision makers. Businesses can experiment and analyze different variables not previously viewed to determine correlations, causation, and predictors, make careful discoveries, and avoid overlaps. This is where industry domain knowledge and expertise come into play. Using available and affordable computing power, storage, and network capacity, businesses can easily analyze more data to see patterns and probabilities hidden in the data. Based on business needs, analytics can be used for descriptive, diagnostic, predictive, and prescriptive purposes. The Internet of Things, sensors, operations technology, equipment maintenance, precision medicine, power grids, shipping, logistics, law enforcement, and precision agriculture are increasingly utilizing the different types of analytics mentioned above to address one or more business problems or to provide solutions as needed.
The need for big data
Big data means different things to different people. Different IT analysts, business leaders, consultants, academic researchers, and standards organizations have defined Big Data according to their views, which include factors such as volume, velocity, variety, accuracy, and complexity. While there is no clear ****ing knowledge on Big Data, their existing capabilities are too big to handle in terms of people, process and technology. People are the hardest part as far as big data and analytics are concerned. There are issues of organizational inertia, lack of support from decision makers, and difficulty in finding data scientists who properly understand the data and business domains being analyzed. Similarly, there is a shortage of big data analysts. Many colleges and universities or accrediting bodies around the world are offering new programs in data science and analytics to meet the growing demand.
Because the field of big data is new and it is difficult to find the right experts, so-called "big data experts or data scientists" are attracted by financial transactions, banks, credit rating agencies, and large financial organizations such as credit card companies. In addition, industry giants such as Google, Facebook, LinkedIn, Yahoo, Microsoft, Amazon, and others are also looking for talent, as they are offering them lucrative salaries, stock options, and better growth prospects. When it comes to competing for the same talent, U.S. federal, state, municipal, and local governments, as well as nonprofit organizations, are at a disadvantage. But some forward-thinking government organizations have managed to recruit some excellent big data scientists.
Overcoming the Talent Shortage Challenge
To overcome the challenge of a shortage of data scientists, many organizations are building a data science team that includes people with knowledge and expertise in big data analytics, as well as industry specialists, such as in IT and business. Together they can complement each other's expertise, collaborate and come up with solutions to business problems. An important characteristic of a successful big data analytics team is the ability to tell stories in business terms and visualize data that requires very little explanation. This is a very specific skill set that requires sales skills to close the deal. These competencies help build the credibility of the data science team or the big data and analytics team in order to gain the support of senior executives and to expand analytics from one business area to another and ultimately to the entire organization or enterprise. These people are the "translators" who can take the results from data analysis and put them into business terms so that the business can understand and adapt. Digital transformation needs to work at the organizational level and become a permanent way of operating. Big data and analytics are an integral part of digital transformation in private or public *** businesses. As a result, many organizations have embarked on a digital transformation journey to unlock the value hidden in big data through analytics. More organizations will follow suit in the future.