The explosion in information technology tools and capability has accelerated. Our information technology environment is moving at absolute light-speed. Yet the back of data integration is broken. 90% of the customers are using 20 year old ETL technology designed to move batch files into the data warehouse at 1 am. It is a manual process. Worst of all, is that some vendors have even put this 20 year old technology in their newly minted 1 year old cloud. Who wants 20 year old technology in a 1 year old cloud?
To add insult to injury, data quality is … “another product.” I humbly submit that without resolving data quality issues and providing for their ongoing maintenance you cannot integrate even two applications, load your data warehouse, consider a future Virtual MDM deployment.
The cause of this is simple. Look at the revenue mix for the two major vendors that control the market for large enterprise. Their business is about professional services – not software tools. They are in business to sell you, their customer, as much professional services at the highest rates you can imagine. Legacy ETL technology is a bonanza to a company that sells your company services by the hour.
Even worse, they don’t offer you the right mix of services and technology. The integration was done to specification but it doesn’t seem to work. I guess we’ll just have to sell you additional consulting to do the data quality clean-up.
The five best practices in data integration break this model. We turn the model upside down. By using advanced data virtualization, and pushing back at the legacy monopoly attached to your wallet, you can reduce your costs and speed your time to implementation.
BEST PRACTICE #1: Use an Automated Metadata Discovery Tool and Data Dictionary. IEEE did a seminal study recently in the area of data integration to understand how the time was allocated. Over 40%+ of the time in any data integration project is spent on discovery. What is the structure of the data? Which field corresponds to others so that we can identify the source record of truth and harmonize it over to the target? Advanced data virtualization can completely automate the discovery of data structures, provides insight into how things relate (semantics) and then automates the integration between the systems. All of this is automated and easy to deploy. You will be able to reduce your time, in this project phase alone, by 75% or more because you use a metadata discovery tool.
BEST PRACTICE #2: Data Quality Must Be Integrated At Every Step in Your Project. Data quality is an essential project component for a small, medium or large company. Not a separate project. Advanced data virtualization technology provides full data quality tool sets, fully automated, with the product. It is the starting point to any data integration, the core to best practices implementation and the ongoing steady hand that keep your data harmonization working into the future. Without good data quality, each data harmonization will reject hundreds to thousands of records that “don’t synchronize.” And this creates tons of additional work.
BEST PRACTICE #3: Understand Your Needs Versus Vendor Architecture. “Tomorrow never comes” is not the right way to deploy critical applications that serve your business. Advanced data virtualization brings a hub and spoke architecture that easily extends and adds the next application to your integration architecture. This is in sharp contrast to the pipe (or pipe and hub for mdm) architectures you must assemble with ETL.
BEST PRACTICE #4: Bring in Tools Suitable For Business Users – No Programming Required. Advanced data virtualization enables your business user to point, click and select to build out integrations. Data transformations between systems come together using a formula builder very similar to Microsoft® Excel® – highly intuitive and obvious. This works the same with relational data, objects, complex API’s with products like SAP®, XML and even NoSQL data sources. One simple object framework designed to work with everything – fast, easy and automated.
BEST PRACTICE #5: Data Enrichment is Easy – Automate It For Business Benefit. Today products like D&B360® and others offer clean data sources for many elements of your important customer data. Yet most of our customers are still manually correcting data at the time of data entry in the ERP system, and then hoping that the support and SFA systems catch up. There is a better way – you can harmonize data directly with a source of data enrichment using business rules. This improves business process and provides strong ROI.