Extreme Makeover — Data Warehouse Edition

by Rick Sherman, Athena IT Solutions

Our high-tech culture suffers from a not-invented-here syndrome. If we weren't the masterminds behind a project, we think it can't possibly be a good one. When faced with a new challenge, we choose the path of revolution, rather than evolution. We break out the latest tools to start building fresh, instead of evaluating and understanding the work our predecessors did.

We assume the previous project team (or management) didn't know what they were doing. We wonder what on earth they were thinking while we ramp up the budget for an extreme makeover -- a total replacement of all their work.

And while there's nothing wrong with having confidence in your own work, there's almost always something to learn from previous projects. As appealing as extreme makeovers are, you shouldn't rush into rebuilding a data warehouse (DW) just for the sake of building it "right" this time.

The bigger the project -- and building a new DW is a big project -- the more difficult it is to justify its budget. And the larger the scope, the higher the stakes and risks for getting it done. Many first generation DW projects failed simply because their huge scope meant they took too long and cost too much.

Keep an open mind -- maybe the project is in better shape than you think. There may be aspects of the current data warehousing environment that can be salvaged and leveraged. Many of the grumblings about the DW are really about a few areas that can actually be fixed. In fact, a closer look may reveal that an existing DW needs a renovation rather than a complete replacement. A renovation enables you to leverage what already exists and to concentrate on what's truly broken. In addition, a renovation allows more time to really understand what the business wants and highlight the gaps in the existing DW.

Meanwhile, a total replacement takes time to select and implement new software and hardware. Some people consider it a fun process, but a preoccupation with tools and technology can divert attention away from the really important and tough issues of data.

Indeed, the data is the tough stuff. Data consistency, integrity, quality, relevancy and currency are the keys to the success of any data warehousing initiative. Data is also the backbone to many of the highly touted enterprise initiatives, such as corporate performance management, customer data integration, master data management and Sarbanes-Oxley compliance. It doesn't matter how "cool" a tool is -- business users have to have faith in the information it provides. Data gets stale and unless we work on an ongoing data quality initiative -- which entails talking to business users, determining what constitutes data quality and consistency, and then monitoring and improving those metrics -- we are jeopardizing success. It means getting into the guts of the data, understanding data anomalies and figuring out what to do. Most of that work involves human interactions and not automated tools, although tools can help with the mechanical tasks.

With a renovation, you can allocate the right amount of time to the right issues and problems. Renovation rather than rebuilding may result in a much higher business ROI, with a better match between your data warehousing environment and your business needs. In addition, renovation means you're not building a redundant DW that would move you farther from that elusive single version of the truth. Many corporations are littered with multiple overlapping DWs, data marts, operational data stores and cubes that were built from the ground up to get it right this time. The bottom line is that you're not creating a DW environment if you keep building silos -- regardless of what you call them. Multiple versions of the truth is an oxymoron.

Have faith in the team that was in place before you. They probably encountered the same data challenges you have today, along with a business user community that imposes tough demands. Leverage the projects that were done in the past. Incrementally renovate what you have. Don't build more data silos. Instead, make the data warehousing environment all that it can be.