Does history repeat itself?

Some of what's "new" in the data warehouse business isn't really all that new. Perhaps I'm showing my age here (I like to think of it as "experience"), but I'm starting to get deja vu.

This month I share some observations about the not-so-new "new" things that are taking up a lot of bandwidth lately.

Business Intelligence Goes Back to the Future - Part 1

by Rick Sherman, Athena IT Solutions

It is said if we fail to learn from history we are destined to repeat it. This sentiment rings true in the IT industry where it appears that a new generation of people - IT, consultants and vendors - and projects emerge every three to four years with everyone thinking that their approach to information access and integration is new or different from their predecessors.

Reinforcing those beliefs is that the industry – vendors, analysts and consultants – have renamed products and techniques since the last generation. Many times the names have been changed because the feeling is that these products and techniques have been significantly enhanced to justify a new name – and a new start. Other times the names have changed because of mergers and acquisitions, or sadly, to “protect the guilty.”

Putting a “fresh face” on something is an effective sales and marketing technique –when there really has been an improvement. However, when the name change is merely cosmetic and people don’t realize it’s the same thing, then nobody is learning from the past.

The high tech industry, like the movie industry, likes remakes and sequels. Evolution rather than revolution is often how we learn and progress. However, many in the high tech industry want you to believe something is new when it really isn’t. Sure, if you have never done it before it is new to you. Also, many of the people promoting the “new” never did it before themselves, so it is also new to them. But even if something feels new to the people promoting using it, it’s not wise for them to ignore the past.

Data integration is a case in point. It’s easy to convince yourself that the reason why data integration is so complicated is because people have never really understood it or had the right tools and techniques. It seems that every four years someone introduces a “new approach” to data integration. But the approaches aren’t new.

Products and the ability to deploy them do improve each generation as servers, I/O, memory, storage and networks become faster, better, and cheaper with significantly more capabilities than the previous generations. But the fundamentals of data management, data quality, data integrity and data integration have been around for decades. And certainly project-management techniques and communication between IT and business people aren’t reinvented in each generation.

Enterprises need information access and data integration now more than ever because of regulation, competitive pressure, expense cutting and revenue growth. We’ll drive ourselves crazy trying to meet this need if we don’t learn from the past and build on what we have accomplished.

Let’s look at some key examples in business intelligence and data integration that aren’t as new as people think they are:

• Multiple “single versions of the truth”
• Data shadow systems are breeding like rabbits
• Business is from Mars, IT is from Venus
• It’s not the tools, it’s the data
• People still don’t understand Data Warehousing
• The Silver Bullet Solution

In this issue we’ll discuss a couple of these areas and will follow up in our next issue with more of the topics.

Multiple “Single Versions of the Truth”

Many companies have built multiple data warehouses (DW), operational data stores (ODS) and data marts (DM); implemented reporting solutions from their ERP and enterprise application vendors that include an ODS, DW and DMs; and have had many groups create – many by accident – data shadow systems. More recently added to the mix are Corporate Performance Management (CPM) solutions that also have DW and DMs built under the covers.

For those old enough to remember, data warehousing started as a concept to bring data from many disparate systems (silos) into one place to get one set of consistent numbers so that the business would know what was happening. Later, this one set of consistent numbers was called the “single version of the truth.” (And, during the Internet boom, the single version of truth related to customer data was referred to as a 360 degree view of the customer. Who said nothing important came out of that boom?)

But what happened with everyone’s attempts? We can never get enough of a good thing. Instead of creating a single version of the truth for the enterprise, multiple groups in the same enterprise managed multiple projects that each created their own single version of the truth…and here we are again – multiple silos with numbers that are not consistent!

Do people deliberately try to create these silos? Probably not, although sometimes politics does result in groups competing to do the same thing. It is far more common that groups simply do not know or understand that other groups are trying to do the same thing. This is especially true when the ERP and DW groups keep charging ahead in trying to support their business people without realizing that their work is increasingly overlapping as the business demands better reporting and analysis from these systems.

Data silos are also spawned when an enterprise buys a vendor solution to solve a specific business problem, but fails to understand or appreciate how that solution needs to be part of an overall architecture. No matter how often you raise the red flag that an enterprise needs to see the forest, not just the trees, they build another silo and are surprised afterwards. And, as I explain next, it’s not just IT projects that are expanding the silos…

Data Shadow Systems are Breeding like Rabbits

With all the data – terabytes and terabytes - available for business users and the cool tools to slice and dice those volumes of data, business people still use Microsoft Excel and Microsoft Access. It’s their primary reporting and analysis tool and often their data integration tool, too. It’s not that business people don’t have BI tools; most companies have several. It’s not that they lack reports generated for them; often companies generate hundreds or thousands of reports repeatedly. And it’s not that the IT department has not gathered volumes of data from disparate sources from around the enterprise into new databases (those silos again...).

Microsoft Excel and Access rule because business users need not only to understand the data and how it was derived. They also need to manipulate it some more to arrive at a report or analysis they feel comfortable presenting to others and using to make decisions. In order for them to get to that point they need to gather data, often from multiple locations, augment it with additional data pertinent to their analysis, and then perform their analysis and create their reports. They feel comfortable with Microsoft Access and, in particular, Microsoft Excel.

The business sometimes feels nervous with the data silos that have been created, so they pull data from them and cross-check the numbers. These activities create data shadow systems – more silos. Business people don’t intend to create these data shadow systems, but they evolve from the processes they use to create information they need to make decisions.

The data-shadow evolution starts when business users pull data from the company’s existing silos (such as a corporate DW). They then transform the data to apply it to their current business needs. They may even transform it in ways that run counter to their initial requirements when the corporate DW was created. Next, they analyze the data in a spreadsheet, such as Excel. Lastly, they use the spreadsheet to create reports, which they distribute throughout the organization.

These systems start with a few spreadsheets and a Microsoft Access database but evolve, rather quickly, to hundreds of Excel spreadsheets and dozens of Microsoft Access databases. Sometimes these systems feed into statistical packages that create even more data. Created by many business groups, these systems are spread throughout an enterprise and, in many cases, rival or even surpass the usage of the DW or BI systems.

As data volumes have increased and data has become more accessible, data shadow systems have proliferated. Many IT groups either do not see these systems or are in denial about them. Business people may not even know they have created these data shadow systems; they just think they are doing what it takes to get their job done in order to help the enterprise. But just like a recovering addict, you first have to admit you have a problem before you can solve it. Both business and IT have to speak to each other, understand what they are doing and try to find ways to make the situation better rather than worse.

Information at the right place, in the right time and to the right people is everyone’s goal, but the pursuit of this goal often leads to a data glut and more data-integration demands.

Next Issue

In the next part of this column we will examine the other issues that keep repeating themselves and start looking at how we can learn from history.