Phone: 650-474-0955


Navigating Big Data

Big Data is an ever-evolving term which is used to describe the vast amount of unstructured data. Published reports have indicated that 90% of the world’s data was created during the past two years alone.

Whether it’s coming from social media sites such as Twitter, Instagram, or Facebook, or from countless other Web sites, mobile devices, laptops, or desktops, data is being generated at an astonishing rate. Making use of Big Data has gone from a desire to a necessity. The business demands require its use.

Big Data can serve organizations in many ways. Ironically, though, with such a wealth of information at a company's disposal, the possibilities border on the limitless, and that can be a problem. Data is not going to automatically bend to a company's will. On the contrary, it has the potential to stir up organizations from within if not used correctly. If a company doesn't set some ground rules and figure out how to choose the appropriate data to work with, as well as how to make it align with the organization's goals, it's unlikely to get anything worthy out of it.

There are three layers of Big Data analytics, two of which lead to insights. The first of these, and the most basic, is descriptive analytics, which simply summarize the state of a situation. They can be presented in the form of dashboards, and they tell a person what's going on, but they don't predict what will happen as a result. Predictive analytics forecast what will likely happen, prescriptive analytics guide users to action. Predictive and prescriptive analytics provide insights.

Presenting the analytics on a clean, readable user interface is vital but sometimes is ignored. Users get frustrated when they see content that they can't decipher. A canned dashboard does not work for users. They need to know what action they have to take. Users demand a sophisticated alert engine that will tell them very contextually what actions to take.

Using such analytics, ZestFinance was able to glean this insight: those who failed to properly use uppercase and lowercase letters while filling out loan applications were more likely to default on them later on. Knowing this helped them identify a way to improve on traditional underwriting methods, pushing them to incorporate updated models that took this correlation into consideration. As a result, the company was able to reduce the loan default rate by 40% and increase market share by 25%.

Unfortunately, insights have a shelf life. They must be interpretable, relevant, and novel. Once an insight has been incorporated into a strategy, it's no longer an insight, and the benefits it generates will cease to make a noticeable difference over time.

Getting the Right Data

To get the right data leading to truly beneficial insights, a company must employ a sophisticated plan for its collection. Having a business case around the usage of data is the first important step. A company should figure out what goals it would like to meet, how and why data is crucial to reaching them, and how this effort can help increase revenue and decrease costs.

Data relevance is the key and what is important to a company is determined by the problems it is trying to solve. There is useful data and not useful data. It is important to distinguish them and weed out not useful data. Collecting more than what is useful and needed is impractical.

Often data is accumulating before a set of goals has been outlined by stakeholders. It is being collected irrespective of any specific problem, question, or purpose. Data warehouses and processing tools such as Hadoop, NoSQL, InfoGrid, Impala, and Storm make it especially easy for companies to quickly attain large amounts of data. Companies are also at liberty to add on third-party data sources to enrich the profiles they already have, from companies such as Dun & Bradstreet. Unfortunately, most of the data, inevitably, is irrelevant. The key is to find data that pertains to the problem.

Big Data is nothing if not available, and it takes minimal effort to collect it. But unfortunately, it will not be of use to anyone if it’s not molded to meet the particular demands of those using it. Some people are under the impression that they are going to get a lot of information simply from having data. But businesses don’t really need Big Data - information and insight are what they need. While a vast amount of data matter might be floating around in the physical and digital universes, the information it contains may be considerably less substantial.

While it might seem advisable to collect as much information as possible, some of that information just might not be relevant. Relevant insights, on the other hand, allow companies to act on information and create beneficial changes.

It is a good idea to set parameters for data collection by identifying the right sources early on. It could be a combination of internal and external data sources. Determine some metrics that you monitor on an ongoing basis. Having the key performance indicators (KPIs) in place will help companies identify the right data sources, the types of data sources that can help solve their problems.

Technology plays a key role in harnessing Big Data. Companies should figure out what kinds of technology make sense for them. Choice of technology should be based on company's requirements.

Data collection is an ongoing process that can be adjusted over time. As the business needs change, newer data sources are integrated, and newer business groups or lines of businesses are brought in as stakeholders, the dynamics and qualities of data collection will change. So this needs to be treated not as a one-time initiative, but as an ongoing program in which you continually enrich and enhance your data quality.

Companies should continually monitor the success of their data usage and implementation to ensure they're getting what they need out of it. There should be a constant feedback stream so that a company knows where it stands in relation to certain key metrics it has outlined.


Companies must always be aware of the risks involved in using data. Companies shouldn't use prescriptive analytics when there is significant room for error. It takes good judgment, of course, to determine when the payoffs outweigh the potential risks. Unfortunately, it's not always possible to get a prescriptive read on a situation. There are certain limitations. For one thing, collecting hard data from the future is impossible.

People and Processes

Big Data adoption often becomes a change management issue and companies often steer clear of it. When a company implements something that's more data-driven, there's a lot of resistance to it.

Like most initiatives that propose technology as a central asset, Big Data adoption can create conflicts among the various departments of an organization. People struggle to accept data, but people also aren’t willing to give it up. To avoid such clashes, companies should make it clear from the outset which department owns the data. Putting the owner in charge of the data, having this person or department outline the business rules and how they should be applied to customers would be helpful to overcome this issue.

These are two good tips to follow: Give credit where credit is due and don't dehumanize the job. Don’t attribute the success to the data, but to the person who does something with the data. Remember that change can't just come from the top down. Big Data adoption requires more than executive support. It needs buy-in from everyone.

Categories: Big Data
(There are no comments yet)
Leave a Comment