This is the second part of a two-part series on BI data integration. See the first part for implementation challenges.
For large organizations with multiple business units targeting diverse customer segments, BI data integration is a must for leveraging enterprise-wide data. At the same time, BI data integration on that scale is very challenging.
You should begin by analyzing your business needs, devising a process pattern, and understanding BI user requirements.
Given below are the top BI data integration challenges, along with tips for tackling them.
1. The big-bang trap
It is tempting to tackle all aspects of BI data integration all at once (big bang), but that approach may not be cost-effective. A piece-meal approach will create fewer challenges. BI data has to be integrated to the smallest level to be useful. Look at each data point individually, create a plan for each, and integrate them one at a time.
Suppose you need to merge data from three applications: a merchandise management system (MMS), a customer database, and a product database. Break it down into individual datasets such as customer information, sales data, financial data, etc., and merge them one at a time.
2. Neglecting automation
Automated BI data integration is less challenging. The less the manual intervention, the fewer the errors and data integration challenges. To cite a retail example, to move data from the MMS to the point of sale (POS) application, we need to generate HTML files. We can then manually upload them to an FTP server, where the POS system can process it at the individual store level. Or, we can put the information in the store folder, where it is processed automatically through a scheduler. The accuracy of the data can be maintained by a system of check-points as discussed in the previous article.
3. Difficulties in drill-down reporting
Data integration is usually the first step towards BI. To support drill-down reporting, data has to be integrated at the deepest level.
In the retail scenario, sales data analytics is done at many levels: product, category, department, store, company, city, region, etc. The best way to integrate data is at the product level, because then the BI tool can automatically merge it with the higher levels as needed. Integrating data directly at the company level, for example, will open up challenges when generating product-level drill down reports. Remember, ‘down to up’ is a better approach in BI data integration.
4. Discrepancies in business data
Data should be actively monitored not only at the data integration design stage, but also at the user stage. Users can point out discrepancies and spot BI data integration challenges that the technology team may overlook. Quality check points should be made available to users also. If the sales data is found erroneous after data integration, users should be able to spot discrepancies in the data in any of the source systems.
>> See Part 1 of this Tip for more on check points.
5. Vendor challenges
Many implementation vendors lack adequate skill-sets to design/implement a BI data integration solution. Seek vendors who understand the technology well and know how to architect a solution. Make the final payment only after the solution is implemented successfully. This will ensure that the vendor tackles all BI data integration challenges that surface during implementation.
Pay attention to finer processes such as extract-transform-load (ETL). To improve data quality, make sure your Extract and Load processes occur at the source level, and not at the target level. The Load and Display processes can then be completed at the target level. Check volume and freshness of data regularly with real-time updates based on business requirements. Lastly, identify all the data points for integration and plan them out before you start the process.
About the authors:
Dipthi Karnad is a senior IT Manager at HyperCITY Retail India. She has over 16 years of experience in IT, of which 13 years are in the Retail IT domain. She has an in-depth understanding of retail business processes and expertise in designing and implementing solutions across platforms. Dipti holds a masters degree in financial management.
Kapil Tulsan works as an IT team lead at HyperCITY Retail India, and has over ten years of IT experience. He has been involved in system requirements study, analysis, design, development, and deployment. He has worked on several projects with multinational companies.
- As told to Mitchelle R Jansen
This was first published in October 2012