Big Data: Challenges, Risks and Solutions

By Nicole Miskelly | 21 July 2015

Many companies within the financial services sector are experimenting with big data and analytics, and although they realise that big data can benefit their organisation, many are still unsure how to use the data to their advantage.

Banks in particular realise that advanced data and analytics technology could provide solutions to some of their biggest challenges such as, retaining customers, keeping up with competition, compliance and tackling fraud. Big data presents lots of opportunities for companies to personalise the customer experience and since reports have shown a decline in additional product purchases from retail bank customers and an increase in abandonment of their primary bank in favour of new bank challengers, banks should be doing everything they can to keep their customers engaged.

Steve Colwill from Velocimetrics, Rupert Brown from MarkLogic and Dr. Andreas Freund from Tata Consultancy Services (TCS) give their own opinions on the biggest challenges companies are facing when it comes to big data, the risks and also the solutions that using big data can present.


Data management

Some of biggest challenges that companies face with big data is understanding how to manage the large volumes of data, organise it properly and then gain beneficial insights from it.

Steve Colwill, CEO, Velocimetrics says: “Two of the biggest challenges we have seen companies face in relation to big data is firstly how they manage the sheer volume of data and secondly how they can later draw meaningful conclusions from it.”

Colwill says that trading is an area that generates an overwhelming quantity of data and there is currently a push in the market to try to retain everything in big data storage solutions just in case it is called for in the future. He says however that, “keeping every market data tick being generated on every market, all of the time often may not be practical over the long-term.  And even if storing it all is possible, being able to wade through huge bulks of unstructured data and find all of the information relating to a trading decision, for example, may be incredibly difficult and time-consuming.”

According to Colwill, analysing data in a fast-paced and ever-changing business environment such as trading is often difficult. “Post-event analysis of data can be especially difficult in environments where business processes are prone to frequent change, as is often the case in trading firms.  In these companies, the way in which the data is analysed needs to be aligned to the business process rules that were in place at the particular point in time when the data was live. This is because the business rules that would cause an algorithm to execute an order based on particular piece of market data today, may differ to the rules that were in place just 6 months ago. So managing this dynamic, and applying the appropriate business rules retrospectively, can understandably present many issues.”

Solutions: Colwill believes that firms can scale down data by identifying which is the most important and only retaining this data. “Managing these challenges requires being able to determine which pieces of data are relevant and important, and as such should be retained, and which pieces are irrelevant. With this approach the volumetric problem becomes much more manageable as only relevant data is retained, so the total quantity being stored radically reduces.”

Rupert Brown, CTO, Financial Services, MarkLogic says that big data needs an architecture and a more agile approach. “Big data still needs to be organised, planned, classified and cleansed. It needs an architecture. A much more agile, iterative approach is necessary, rather than an approach that focuses on a big design up front.”


There are currently a number of regulatory restrictions that are impacting the effective use of big data. Banks are under pressure to comply with the Basel Committee on Banking Supervision (BCBS) 239, which outlines 14 principles around risk data aggregation and risk reporting. According to Brown, BCBS 239 is “focused on understanding the provenance, lineage and classification of data and is probably the most significant regulation.”

Another regulation having an impact is the US Securities Exchange Commission’s Regulation Systems Compliance and Integrity (Reg SCI) which according to PwC, is a comprehensive and wide-reaching regulatory regime that requires certain market participants that are key to the functioning of the US securities market to “have robust technology controls and promptly take corrective action when problems arise,” and which Brown believes, “will have a significant impact on the operating procedures and management of the datasets.”

Solutions: Although regulations can be hard to comply to, big data and analytics can often help provide a cheaper option to paying compliance costs and if used right, Freund says that “patterns that can lead to violations can be detected through predictive analytics before an actual violation occurs - predictive compliance - and, moving further upstream, Big Data can recommend the best actions within a particular context to prevent compliance violations.”


Flawed Results

When it comes to analysing big data, if companies do not understand the data, and misinterpret it, then they risk generating unreliable results. “One of the biggest risks is the storing and subsequent future analysis of unstructured data in a way that generates flawed results,” says Colwill.

Freund says that a lack of understanding about what big data analytics can and cannot do often leads to “an over interpretation of results” such as a confirmation bias, which is when people actively search for and favour information or evidence that confirms their preconceptions while ignoring adverse or mitigating evidence.

According to Brown, “Many people think that you can just create a data lake and the answers will magically appear.  The time and cost of managing and moving big data sets around cannot be underestimated.  Without a storage architecture and taxonomy the lake soon becomes a swamp.”

In order to make sense of data, organisations need to understand the complex relationships that exist between big data and how these complex relationships may change over time, says Colwill. “The issue is, doing all of this retrospectively depends on a lot of things being applied exactly right. Get one thing wrong and any assumptions drawn will be flawed.”

Data Security and Privacy

Data theft is a growing area of crime, and as shown in recent high-profile hacks, the bigger your data, the bigger opportunity it presents to hackers. Because of this, companies need to make sure that customer and client personal data is secure from external threats, as well as ensuring that the sensitive information that they have is protected from internal threats. Companies that fail to comply with data protection laws could also find themselves footing the bill for expensive lawsuits or time in prison.

Solutions: Freund believes that privacy will “become more and more a thing of the past as our digital footprint increases unless we transition to a decentralised world based on blockchain that has strong encryption built in.” Security however, he says is another matter. “Big Data per se is not a security risk, unless it is used to systematically find human behavior patterns around computer systems targeted for a hack that point to vulnerabilities, for example, individuals reusing the same password for not only private but also enterprise use as is often the case.”

What else should companies be doing?

Organising a data strategy

Having a well-organised data strategy is one way for firms to make sure that big data is being handled properly. Freund believes that organisations should be organising their data strategy the same way that they create and revise their overall business strategy. “Pick your preferred method be it BCG Strategy Framework or Hoshin-Kanri, just do it. If organisations do not realize that data is their most valuable asset and that, therefore, their business strategy needs not only to leverage but also aim to monetise their data through new products and services built on data, they will go out of business.”

An even bigger question that many companies struggle with is who should be responsible for creating a data strategy. “Data is everyone's responsibility! Therefore everyone needs to get involved ... eventually. To start, figure out the most important questions you need answers to and then pull everyone together who has a stake in finding answers to those questions including IT plus a data scientist and people with Big Data technology implementation experience, no rookies!” says Freund.    

Brown believes that the key members of staff that should be involved in creating a data strategy should always be those at the top. “CEO and Group executives. It needs to start at the top and then cascade down, with partnerships between business and tech teams at all levels and stages of the process. The boundaries between CIO and CDO also need to be clear.”

What are new players doing differently?

In recent years, start-ups have been disrupting the financial services space with innovative technology offerings and the flexibility needed to respond to market needs. Colwill believes that in the area of payments processing, new players focus more on the customer experience. “If you look at the payment-processing environment I think one of the very interesting things we are witnessing from some of the newer players is their strong focus on the individual customer and their experience. The more traditional players are now looking at how they too can use their data to even more effectively understand situations from their customer’s perspective.”

Colwill says that in a technical outage situation, by being able to effectively analyse big data in payments, firms are able to quickly understand exactly which payments were being processed by the impacted system at the time and the client’s that submitted them. Which means the company can proactively contact impacted clients, potentially before the client even realises their payment has been delayed.

The Future

Colwill believes that for the data being stored today to be of value tomorrow and even in a year’s time, it is really important that what is stored can be meaningfully analysed. “I strongly believe this comes from storing data as complete chains, tying together all related information so the data can be easily followed. The continual improvement of the quality of data stored I see as the next step in this evolution,” says Colwill.

Freund believes that in the short term and with the Internet of Things (IoT) on the rise, better visualisation and simpler data integration tools together with streaming analytics and automated model building will be hot.

IoT technology enables objects to communicate with other objects and analysts predict that by 2020, between 26bn and 50bn IoT units will be connected to the internet. According to Brown, IoT creates big data sets, often with very significant geospatial content.

Freund believes that “analysing IoT data using Big Data will be critical for all businesses to understand business opportunities as well as business threats in real-time as we are going into the Zeta Byte era of the internet where change is not only constant but exponential.”

Freund also says that as Artificial Intelligence (AI) becomes more central to what and how we are doing things in our lives over the next five years, Big Data will start to fade into the background and become a key infrastructure layer.


Become a bobsguide member to access the following

1. Unrestricted access to bobsguide
2. Send a proposal request
3. Insights delivered daily to your inbox
4. Career development