CDOs, CIOs, enterprise architects and conscientious data management executives have increasingly complex set of choices and decisions to make. Information that gives clarity on forward path is invaluable in the digital age. At the heart of this is data, be it Big Data or traditional column stores, and managing that strategically is key. In this article we take stock of enterprise data management challenges; and a solution roadmap which improves the ability to create a successful enterprise data management framework that supports the organisations goals of accurate business insight and competitive edge.
While vendors love to talk about simplicity and efficacy products; hardware, setup and installation considerations are just the tip of the iceberg. The huge amount of customization required to fit the individualistic needs of a organization can require far more effort than expected, negatively impacting the success of an enterprise data management program, complicated by misconceptions around Big Data. Whether there is a need for distributed multi-node file systems in non-column store databases can be evaluated best by looking at the need of routine repetitive tasks that can be heavily parallelized along with the volume and variety of the data..
On the other hand a greater time to market, higher overheads and the challenge of getting the right individuals with the proper skill set brings down the overall appeal of a build approach. If open source systems can be taken up as data stores and proprietary data intelligence algorithm built on it, the TCO of such a solution is quite lucrative. But scalability, future integrations and migrations are important considerations. Composite architecture is the de facto choice for a large enterprise. HDFS processing for monitoring real time payments activities has strong business value, while RDBMS makes for a good choice in business functions such as depot management, accounting and reconciliation.
With outsourcing there are the immediate advantages of cost, overall productivity improvement, reallocation of valuable man power, absence of the overhead of management and faster time to market. But there are huge questions over reliability, security and liability. There is often proprietary data and sensitive data that is difficult to put on the cloud. But today’s cloud based solutions offer a host of advantages and a security innovations that are a fusion of homomorphic and order preserving encryption, and trial results are promising. Peak demand, risk evaluation and data centre operations costs are other key considerations.
This evaluation leads us to think out of the box. Rather than focus on a particular approach independently, it should be evaluated in the context of the organization needs and vendor capabilities. Enterprise data management is the right way to think strategically. For implementation this must be approached in a phased manner where operations responsibility is gradually transitioned to the vendor, thus attaining a cost efficient steady state mode in the very foundation of the architecture for business insights.
With the benefits of outsourcing kicking in, more units can be brought under the fold of the enterprise data management program. The internal data management team can hand over most of the BAU tasks to the vendor while reallocating valuable managerial talent to other critical tasks like deriving the forecast for the future with the confidence of a smoothly humming robust machine churning out reliable data.
Irrespective of the choices, data stewards have to address two significant challenges:
1. Data acquisition, transformation and management
2. Data publication and distribution to different systems
When architecting an enterprise data management solution, both of these aspects must be kept in mind. Typically the focus is towards data acquisition, scrubbing and consolidation. This short sightedness is costly given that the essence of providing a single data hub for use by various systems of different business units. This is why publication and distribution of data from the hub is of immense significance and must be architected in the solution blueprint.
In summary, post a decision on buy/build/outsource or partner, five basic tenets can be applied for enterprise data management success:
Scenario A: Still in Concept or Initiation State
Scenario B: Execution has started and issues are cropping up like weeds
By Joy Mukherjee, bobsguide contributor