With a considered strategy the truth in big data is not that elusive

Very recently, when it has been extensively reported in the media that Microsoft researcher David Rothschild has accurately forecasted this year’s 20 Oscar winners out of 24 with the help of his predictive model, our paths have once again crossed with big data in a fascinating way. But then, it’s a glimpse into a smorgasbord of transformational capabilities that it brings. For businesses, the lingering buzz around big data doesn’t ring hollow as they are unlocking unprecedented value by applying it. An Accenture survey has captured this pervasive sentiment; 94% of companies have admitted that their experience with big data has proved to be an exact match for their expectations. As such, related investments are coming in thick and fast. Capgemini Consulting predicts that three years hence organizational spending on big data technologies will end up at $114 billion.

On this splendid note, let’s have a retrospect of major developments in the concerned space in 2014. Also, we will look at some of the success stories, and the hurdles that have got businesses in a sweat. Finally, a short lowdown on the best practices that make the tryst with big data a meaningful and plain sailing affair.

Cloud is becoming the new standard destination for data

The mass exodus of companies from a physical set-up to a cloud based one has been a significant event last year. This has happened primarily because a host of data processing technologies are now fully operational on cloud platforms. Sensing the opportunity, leading vendors have spared no effort to drive companies towards cloud. Offerings such as Salesforce’s Wave, IBM’s Bluemix, or Amazon’s Redshift have gained a healthy stream of customers. In fact, a hybrid architecture that combines the cloud and on-premises will form the backbone of data crunching in near future.

Hadoop in the spotlight

The open-source and distributed big data solution Hadoop has emerged as the default analytic framework for a large number of enterprises. In a sense, it is the data OS for them. Moreover, products that promise SQL-on-Hadoop are making life easier for enterprise users. These are light on company coffers as running them do not require the services of experts with specialized knowledge.

Increased demand for predictive analytics

Storing data in Hadoop is not the end in itself. It’s just the beginning, as analysis of the data stacked there is what generates valuable insights. An array of advanced processing engines in 2014, whether open-source Spark or other commercial alternatives that are designed to manage data with multiple attributes, have enabled companies to get real benefits.

Options galore for learning

With more and more companies deciding to dabble in big data, demand for skilled data professionals has soared in 2014. This has encouraged education providers to come up with various industry-ready training programs on the subject.

Data lakes cutting a wide swath

Data lakes are basically repositories that can store data in native formats for parsing at a later time. Since the concept is an effective tool in combating data fidelity and integration issues, companies are finding it useful.

Big data has hit the big time

Big data has a lot of use cases in an enterprise environment. According to the Accenture survey, 85% of users think that it has phenomenal power to change the workings of a business. They have identified customer relationship, product development, and management of business operations as the areas where the impact of big data will be felt the most. Almost replicating this tone, a QuinStreet enterprise report says, to 72% of companies the biggest advantage of big data lies in fast and spot-on decision making.

Here’s a few instances of successful implementation of big data:

  • Fashion retailer Macy’s relies on big data to adjust prices of more than 70 million items in real-time.
  • Pharmaceutical company Merck is leveraging big data to streamline vaccine production.
  • Financial services giant American Express is mining big data for loyalty prediction.
  • Through the self-developed semantic search engine Polaris, Wal-Mart is pushing up its online conversions.

That big data is a boon to enterprises is no secret. However, the process of adapting to a big data-driven culture is not free from challenges.

  • The scope of big data is not restricted to a particular department rather, it’s a company-wide function that demands effective integration of various data fountainheads. So, firms that follow a siloed architecture are losing out in the game.
  • Lack of right talent is a big concern. From the implementation stage to extraction and interpretation of insights, every part of the big data process must be handled by people with domain expertise. Ineptitude can make plans go wide off the mark.
  • As the number of security breaches has spiked in recent times, protection of data sanctity is a top priority for companies.
  • Trying to manage big data through legacy systems is hurting companies. They must opt for the latest set of tools.

Practices that move things in the right way

The best anodyne to fight the pain generating from the issues discussed above is to develop a dedicated big data strategy.

To conclude, a ‘separate the wheat from the chaff’ approach is needed to discover the truth in big data i.e., enterprises must clearly understand what information to keep and what to discard.

  • Kick-start the initiative by building a stable governance framework that will act as the mainstay. This should include carving out a top level leadership role for data, establishing benchmarks for measurement, and picking up the use cases
    .
  • Next, reflect on the combination of technologies that will exactly serve your purpose because there is no one-size-fits-all.
  • Create a big data-ready talent pool. Do conduct skill development programs at regular interval and don’t hesitate to harness external resources when needed. You may also consider forming tie-ups with academic bodies to hire prospective recruits.
  • Lastly, but in no way least important, make sure the big data platform you choose must be agile so that you can incorporate the latest changes quickly.