Financial enterprises have long plumbed masses of detailed business and financial detail to identify trading and other opportunities, but the new regulatory climate in the wake of the Great Recession has created transparency and risk mitigation applications for big data analytics in this area as well.
This is only one of many uses for big data analytics, which are also being applied in areas as disparate as aggregation of social media activity data for e-commerce applications and in political analyses of campaign donations and of demographic and voting patterns. Big Data analytics are generally distinguished from traditional business intelligence and enterprise resource planning software applications by large volumes of data in an array of non-traditional formats that traditional data warehouses may not be able to easily accommodate.
The transparency and reporting requirements of Dodd-Frank have made Big Data analytics increasingly relevant to regulatory compliance, but useful applications also exist in risk analysis and mitigation as well as in portfolio analysis and management.
Clearly, grappling with the storage requirement pain points caused by the data's disparate formats will be necessary in order to harness the information it represents, but even before that, it is important to implement an organized and rigorous framework for implementing a big data analytics system, to ensure, among other things, that the information the enterprise already has is properly identified and leveraged.
Remember, for example, to include transaction, supply chain and log data from your web servers, and any online services to which you may subscribe, as well as call and usage detail from employee smartphones. Social media analytics, government statistics and other public records, sales and product data and CRM entries should also be considered.
Once you have a detailed inventory of the data that you already have available, take a look at its characteristics and consider what would be required to link it to pertinent data from other sources, in other formats, and based on a different set of assumptions. The question then becomes how you take that untidy pile of data and parse through it to discover the opportunities and threats that your organization is facing but may not yet have noted. Remember, it is just as important to identify the things that aren't working or that are wasting resources as it is to identify sales opportunities or new logistical efficiencies.
Your implementation planning should also take into account the projects and initiatives that are currently underway but not yet complete; the whole point of big data analytics lies in its ability to pull together information you didn't know you had to tell you things that you didn't know that you needed to know.
If your enterprise does not have a person, or better yet a team, designated to improve the quality of your organization's data, it should. The team should investigate questions like whether it should add new data sources to the process, and how to annotate trend data to show the effect of outside events such as natural disasters or computer downtime, to prevent their effect on operations and sales from being factored into analyses as normal variation. Similarly, the sampling and analyses associated with customer or vendor surveys should also be documented. Finally, don't overlook the possibly invaluable external data you may have accumulated in a web analytics account or on web server logs in the cloud.
