What happens when new technology can’t keep up? At MarkLogic’s World conference, CEO and President of the enterprise NoSQL big data platform provider, Gary Bloom, spoke about how IT can occasionally become a hindrance rather than a solution and it is important to understand that all businesses want a 360 view of their data, in whatever capacity it may exist.
A major challenge that companies are facing is extracting the data that is stuck in isolated systems, or silos, because the technology has fallen behind. Bloom explored how a few years ago, 80% of data is unstructured, but businesses are still struggling with both unstructured and structured data because of the silo problem.
Bloom stated that “every industry has a different reason for a different problem that relates back to the database.” A solution to this problem has to be found in an operation where there is a unified, actionable and 360 view of data and in the eradication of data that is spread across disconnected databases.
“We’re solving some of the most complex data challenges today,” Bloom said as we are working in an industry where new data is continuously emerging. In order to extract, transform and load (ETL) data on relational databases, businesses need a lot of time and a lot of money, which are luxuries, especially when you are a bank with an impending fine.
Prior to the financial crisis of 2007, banking departments were told by regulators to operate as separate entities, but when the crunch hit, the same regulators asked for all data to be aggregated. This was of course, difficult, Bloom explained, as the data was stored in isolated systems “that were not designed to talk to each other.”
Another technology challenge exists in how the human IT department needs to work quicker to ensure that tasks are completed on time, but it can take a few years to aggregate data and this is why an ETL system does not work. The alternative that Bloom offers is the NoSQL platform that is “an operational and transactional database that integrates data better, faster, with less cost.”
80% of time is wasted by data scientists just wrangling data
60% of the cost of data warehouse projects is on ETL
$36 billion was spent on creating data silos with legacy technology in 2015
The reality is that businesses cannot lose any data, but it is also a fact that it is difficult to piece together all your data to get a full view of the customer, as David Gorbet, senior vice president, engineering, at MarkLogic said. Gorbet used the analogy of Morpheus offering Neo the red pill in The Matrix to highlight how companies have the opportunity to embrace data integration in an innovative way.
Gorbet also encouraged the audience to step away from ETL and only harmonise when necessary so that source content can be preserved and queried. “It’s only when you bring these things together with the meaning intact that the magic happens,” Gorbet said. However, alongside this, as always, compliance with regulation is a process that is mandatory for companies.
With the multitude of requirements that need to be completed due to increased regulatory reporting, risk management and investor transparency obligations, it is of paramount importance that capable humans and systems are put in place. Michael Henry, Principal, Advisory, KPMG LLP New York, explored how this is untenable and with the number of solutions that are available, “it would be foolish to continue to attack this problem with humans as the digital labourer is far more capable.”
Coming soon to bobsguide: An interview with Michael Henry, KPMG LLP on the digital labourer and the future of data.