If you're attending Gartner's Business Intelligence Summit March 6-8 (in Chicago) I'll be in the Solution Showcases Monday and Tuesday, and delivering a session on Monday. I hope you'll stop by and say hello.

I'd like to extend a special welcome to those of you who subscribed after last month's DCI conference. We had a lot of fun in Chicago reconnecting with the data warehousing crowd and meeting some new faces.

Have You Met Your Data Shadow Systems?

by Rick Sherman, Athena IT Solutions

Do you really know how your company’s business groups are getting their hands on the business information they need to operate the company? Do you want to know?

Picture this typical scenario: the finance group is cranking out reports to examine business performance. They’re looking at revenue and expenses versus budget and forecast. They’re figuring out what is selling, who is selling it and how much it costs to produce it and deliver it to your customers. And they are doing all of this in spreadsheets…dozens, maybe hundreds of spreadsheets.

Look closer and the plot thickens! Dozens and dozens of Microsoft Access databases are being used to extract data from your ERP and enterprise applications, and even your data warehouse, and transform that data for use in their spreadsheets. Despite all the BI tools available, your company is using personal databases and spreadsheets for data integration!

Does this sound familiar? Not only is the finance group doing it, but marketing, sales, and other organizations are also using Microsoft Access and Microsoft Excel to gather data from across your enterprise, transform it, report on it and analyze it.

It’s the same story at Fortune 100 companies and small-to-medium size firms. All the business users are doing it. Including yours. And there’s a name for what they’re creating: data shadow systems.

Business users build data shadow systems to answer the business questions that the enterprise applications, data warehouse or reports fail to answer for them. They’re filling a gap in the services they receive from their IT departments. Users may not want to get the information this way, but they don’t see any alternative.

Data shadow systems satisfy business groups’ need for relevant business information – exactly what they believed the elegantly designed vendor-built application and IT business intelligence systems would do for them. As data shadow systems evolve over time, they encompass more and more information, and increasing numbers of business users come to depend on them.

So, data shadow systems give business groups what they want and everyone is happy, right? Not quite. The fact is that business users do not want to spend so much time creating these systems. Nor should they. They should be spending their time gaining a better understanding of their business, not wrestling with technology.

Because dealing with technology is not what business users do best, they cobble data shadow systems together without an overarching design. Each addition gets more difficult to implement and more costly to maintain. And when data management principles and disciplines aren’t followed, data consistency and integrity suffer. Yes, data shadow systems often fulfill their business’s need, but they do so in a very costly manner that uses too many resources, and sacrifices data quality.

What needs to be done? Leverage the inherent business value of a data shadow system while rebuilding it to take advantage of better design techniques and more appropriate tools.

It’s important to note that what does not need to happen is a two-year IT project that implements the best-of-breed technology, standards and practices while paying for of the most expensive products in each software category. This over reliance on costly, “silver-bullet” technology is the reason why we keep hearing about a 70% failure rate in data warehousing projects. It is not that data warehousing is a failed approach, but rather these mega IT projects generally don’t meet expectations, nor do they provide a reasonable return on their huge investment.

Rebuilding data shadow systems is the right thing to do to ensure consistent, quality information for running a business. How one approaches this task, in a way that maximizes value and ROI, is a story for another day.