Big Data is becoming more and more important in a whole hosts of industries, with marketers, retailers, insurers and the finance sector all making use of the growing amount of data available to dictate decisions.
However, the concept of Big Data is a relatively new one and companies are keen to safeguard the breakthroughs they are making in the use of the approach. While this makes sense in the short-term, many argue that withholding information from others within the same sector may actually be doing harm. This is particularly true within the finance industry.
Since the financial crisis, financial institutions have been trying to find a way to collate and share data across the board, in a bid to monitor data generated in relation to complex securities, which have, up until now, been spread thinly across a number of individual organisations.
Attempting to make data more readily available for the finance sector is a sensible approach, as reporting systems have proved to be unreliable in the past, with the issues leading up to the financial crisis going unnoticed by many.
The Bank for International Settlements published a report in 2013 that demanded the creation of a joined-up database within the financial sector, in a bid to increase the ability of regulators to monitor the entire global network of over-the-counter derivatives at a detailed level.
According to the Financial Times, the need for increased and more stringent monitoring of finance practices and the use of data led to the US Congress creating the Office for Financial Research (OFR), which makes use of Big Data and other technological innovations to track financial flows and monitor developments.
Last month, Richard Berner, head of the OFR, told Congress that the financial crisis highlighted a number of serious issues in the understanding of the vulnerabilities of the global financial system and the data required to measure them appropriately.
In Europe, the desire to make use of data to monitor finance systems is also growing, as indicated by the recent announcement of a collaboration between First Derivatives and Pivotal Software.
The two companies are hoping to tap the increasing reliance of capital markets on big data by supplying "robust and adaptable" technology.
"From structured to unstructured, time series to transactional and reference data to metadata the financial services industry requires instant, flexible access to all data types. Pivotal One provides that common access, and First Derivatives provides consulting services on how to utilize it in the development of next-generation financial applications," First Derivatives said in a statement.
Despite the progress being made and the measures being put in place to ensure financial data is shared across the sector, recent scandals have thrown a spanner into the works. Spying scandals have made politicians and bankers wary of collating information in a central database.
Kay Swinburne, a British MEP who sits on the chamber’s influential economic and monetary affairs committee said that the ability for financial supervisors to get a holistic view of finance systems is in danger.
"The NSA scandal has caused many to reassess the risks of their personal data being shared across borders.”
Resolving these concerns may take a great deal of time and the longer it takes for those in the finance industry to get past their worries, the longer the financial system will remain unclear.
By Claire Archer