Data and Analytics - The Dark Secret Nobody Talks About

 

There is a dark secret about the data warehousing industry that no one ever seems to acknowledge, and it has nothing to do with technology.  Sure, there are all sorts of technical reasons that can be blamed as far as why a data warehouse did not deliver the business value that was promised, but that’s not the whole picture.  Surprisingly, even data warehouse implementations that were consider a success from a technical standpoint can be deemed a failure by the business users and this is where the real problem lies.

Why can that happen? You’re about to find out…

Before we get there, let’s talk about computer systems. In my mind, there are two broad classifications of computer systems: the must haves and the nice to haves. The must haves represent systems that the business would crumble without. These are systems like financial systems where you enter vendor invoices so that they can be paid, or you enter customer invoices so that the customer can be billed and they can pay you. Manufacturing systems that help you build and ship product.  The list goes on and on.  Without systems like these, it would be very difficult, if not impossible, to run your business efficiently. Some would argue that without these systems, your business would die.

The nice to haves are systems that could be considered “optional”. These are systems that if they are not in place, the business will survive. The business may run a bit less efficiently, but there’s not a worry that the business would crumble. In many industries, data warehouses are considered a nice to have. There are a few industries where without them a business would be extremely limited, but not a whole lot of cases where a business would die without one.

In my years at DecisionPoint, I experienced a lot of sales situations where we would get really close to closing a deal, but the customer would lose their budget, the money they had budgeted would go for another IT priority, or something else of that nature. If the customer walked away from us, the business was not going to die. Sure, the company could greatly benefit from the data warehouse, but there was no risk to doing nothing. No money spent, no risk of failure, etc. I think that’s what makes selling data warehouse solutions such a tough business. Anytime the customer has a “do nothing” option, it’s always something to worry about.  There must be a motivator or compelling reason that causes them to not choose the “do nothing” option.

There’s another “do nothing” option that never gets mentioned by IT shops, consultants, or vendors. That “do nothing” option lies on the other side of a successful implementation of the data warehouse. Let’s assume that you went through a lot of trouble and hassle, but were able to get the data warehouse up and running with end users actively using it. You think all is well and good, but there doesn’t seem to be any business value that comes from the data warehouse. All of the potential business value (i.e. increased revenue, reduced cost), etc. just doesn’t materialize and you wonder why…

Believe it or not, the answer involves something that is hidden within the company and can be best expressed using a bit of Psychology. What?  Hang with me…

Anyone that has been through or knows about counseling should know the answer to this question. If you’re going to a counselor, when do you start to get better? The answer is you start to get better when you decide to get better. The counselor can talk to you for hours and make many suggestions and recommendations. However, if you don’t decide you want to get better, you won’t try any of the recommendations. The counseling sessions will simply be an hour or so of your time talking to someone. There is no benefit, and you’re probably wasting a lot of money.

Now, let’s spin this in the direction of the data warehouse. We’ve done everything that needs to be done and the users are using the data, but nothing changes in the business. It’s quite possible that the data warehouse is providing valuable information, but the users are too afraid of what the data tells them. Just like a counselor, the data can tell you a lot of things (good and bad) about what’s going on in the business.  Unless users can openly recommend changes to a management team committed to changing, nothing will happen.

Your data warehouse is just like going to a counselor…you’ve spent a lot of money, but nothing changes because the organization has decided that it doesn’t need to “get better”.

In all my years working in data warehousing, it amazes me how often this happens.  The data can tell you exactly what you need to do, but the organization just can’t accept that it needs to change.  Sure, you will see some moderate benefits from the data warehouse, but if you really want to hit the home run, the organization needs to know that it will need to change in some way.  And, honestly, there are a lot of organizations that hate change…not because of change itself…but because of all the things that could go wrong.  I’m not saying that you don’t need to be cautious, but you also need to accept reality even when it’s a bitter pill to swallow.  A data warehouse is the first step to change. It tells you how things are so you can see where you need to go. Getting there involves change, and in some cases, change represents a bad thing. Don’t rock the boat…you might get in trouble or be fired. Willingness to accept and use the data as a “business weapon” is the only way to succeed.

Here are some examples that I experienced first-hand…

The names are left out to avoid any kind of aggravation I might receive from the people that were involved. Many years ago, I was helping a customer in the Northeast United States implement a data warehouse on their finance and accounting data. It was about 8-12 weeks of me flying back and forth across the country to complete the implementation. Like any project, we had some bumps along the way and worked through them.  However, the bigger problem was that we ran into a brick wall on the business side of things. We started validating the data and get a feel for what it was going to show us.  As we dug deeper, the user looked at me and said: “You have to turn this off!” After stumbling over my words in disbelief, the response went like this: “Your numbers are correct, but that’s not what the CFO believes the business looks like…” We could have fixed any technical issue that came up…that was the easy part.

What was not fixable was the unwillingness to go to the CFO and present the information so that they could figure out a way forward.  The amount of change required to make thing better was far more significant than they anticipated.  No amount of coaching or discussion from me was going to change that.

In another scenario, I was working with a company that had retail stores that rented movies (very long time ago). The thing that you quickly learn about retailers is that when it comes to change, they don’t mess around. If they find something that will eliminate costs or increase revenue, they’ll do it…no questions asked. However, it does get a bit cut-throat as there is “no mercy” in what they do. They may make a change in one day that puts 100 people out of work, and their attitude is “so be it, we had to do it for the good of the business”.  As a person who cares deeply about others, that can be difficult to see

We weren’t even live with the data warehouse, but they insisted on taking a look at the unprofitable stores.  Literally, for any store that was “in real trouble” (profitability) the process for shutting them down started the next day. No questions asked. This was long before they were even actively using the data. They saw a problem. They fixed it. They got the outcome they had hoped for, but I still wrestle with the way in which it was done.

At the same retailer, much later in the implementation we helped them find something else. We were looking at vendor invoices and payments along with trends in expenses.  For most of the year, the trends looked like what they were expecting.  When we got to the month of December they were shocked. The vendor invoice volume from November to December quadrupled. Obviously, they were determined to find the source of this phenomenon and dug deeper. It was a long process with a lot of details. 

To make a long story a bit shorter… They discovered that when they would hire a store manager, the manager would get a bonus each month related to the profitability of the store. The managers had creatively figured out that if they didn’t turn in their vendor invoices (i.e. expenses), the store was more profitable. For the first eleven months of the year, the managers wouldn’t turn in any store expenses to the corporate headquarters. When December rolled around, they would send in all of the vendor invoices they had collected over the year. This created an accounting nightmare.  In addition to the original invoice, the stores were also getting late notices on those invoices. When all the invoices were sent to corporate headquarters in December, there were cases where vendors were paid multiple times for the same invoice.

What started out as what they thought was a simple problem was costing them a lot of money. And, as you might expect, the bonus program was changed and the problem disappeared.

The final example involves the risk management department at a grocery store chain. By far, this user and organization get the “most creative use of data” award from me.  When you insure hundreds of vehicles to get groceries from a warehouse to a store or between stores, the risk management department spends a lot of time looking at insurance policies.  The sheer number and cost of insurance plans to cover the delivery vans and trucks was mind boggling, and they were always looking at ways to reduce insurance rates without loss of coverage.

Unbeknownst to me, one of their risk management employees did some analysis using some vendor invoices related to truck tires.  OK, that’s interesting… what I didn’t know is that looking at truck tire purchases allowed her to compare what they were spending on new tires for the vans in each region of the country. Beside that, she compared the insurance rates they were paying on each van in each region. It turned out that the regions that spent more money on tires for the vans had vans with fewer accidents and the insurance rates were lower. Using this information, the user was able to institute a corporate wide policy on how often the vans would get new tires, so that all regions were doing the same thing. Once the tire policy was changed, they began to save millions of dollars on their insurance policies. The user had the will and authority to do what needed to be done, and it made a huge difference.

What’s the moral of the story? It’s very simple. When it comes to a data warehouse, the business only improves when the users are willing and able to act on the information that they see. If they are unwilling or unable due to a management resistance, the data warehouse will die a slow painful death. Potential only goes so far…you must be willing to use that potential before anything changes.