Link This | Email this | Blog This | Comments (0)

The Danger of Surveys
September 25, 2008

I was just leaving the Blockbuster Video store when I was approached by a young lady with a clipboard.  She looked like she had not been having much luck that morning, so, given I was not in any sort of rush, I stopped to answer her questions. 

She asked about my exercise regime, calorie intake and alcohol consumption.  As I considered my responses, I experienced a twinge of guilt, as many of the questions centered around a healthy lifestyle.  This youngster was clearly fit and healthy herself, I didn’t want to disappoint myself or her. 

So, let’s say I was not as truthful as I might have been.  I considered the last week, when I had travelled to Chicago and Boston and fine-dined on both occasions; not to mention the wine.  I rationalized to myself that this was not a “normal” week, and as such I should regard these events as “outliers” to be excluded from my self-assessment.  The fact that I had carried my running shoes and shorts in my luggage, but somehow had not found the time to dust them off for an outing, was surely out of character. 

This is not exactly a deception – I would rather phrase it as a “softening” of the harsher truths that I feel righteously entitled to – since I had given up my time to the surveyor for free.  What’s more, I thought, since the sample size was probably enormous, my answers and their accuracy would not skew the outcome.  My conscience was clear.

On reflection, I later managed to rationalize a viewpoint from which my answers have the ring of truth to what I presume to be the intent of the survey.  How so?  Because my answers reflected the “me” I fully intend to be in three or six months’ time; in essence, my future state as I imagine it to be.  Much like a quiz, I believe I have calculated what the right answer “ought to be.”  Surely this is what the researcher was getting at? 

And What About the Truth?
I would be being truly unkind on myself to step back and observe that this future state has been three to six months away, for many years…perhaps even a decade.  Ouch.  Just as well the survey was not an important piece of research. 

And there lies the problem with surveys.  We all know that our answers are a reflection of ourselves, and it is in our nature to “put our best foot forward.”  But our “best foot” is not often the truth.

And Why Does It Matter?
I reflected on this story when I read a survey of food makers stating that the Overall Equipment Effectiveness (OEE) in most food plants today is 88 percent.  This is completely unbelievable—period.  I know from our own benchmarking work with well over a 150 plants that this cannot be true.  Indeed, I assert that it is not only unbelievable, it is dangerous.

The phenomenon I described in my story is aligned with the OEE statistic in the report, regularly adjusted from plant to plant to reflect “local” conditions and to be “just a little more forgiving” (some with “breaks”, some without, some with certain changeovers, some without, some with scheduled downtime and some without… some collecting most downtime incidents, some not, you name it).  Certainly, these food producers were reporting their desired future state, not their true current state.

It’s Not Life And Death, This Is Much More Serious 
There is—and should be—a growing concern about the difference in perspective between survey-taker and survey-respondent.  The survey taker reports her findings with all seriousness—she is looking for meaningful trends on which to report and to influence action.  Yet, the survey respondent answers the survey hopefully and even trivially and his answers are completely unverifiable.  This is not a good basis for policy making in business .  I’m sure you’ve heard the apocryphal quote “This is about money, so come-on, it’s not life and death…this is much more serious!” 

This phenomenon is dangerous because if we believe that optimistic survey data is true, we eliminate our belief and motivation to focus on and act on significant improvement.  For example, if I believe that I am already at my desired future physical condition, why bother to dust off my running shoes!

 An OEE of 88 percent, if it were true, would leave little, if any, real need or room to improve.  Yet, from recent benchmarks of over 150 locations that I have been reviewing, the gap between “reported” and “actual” output performance of U.S. and UK food factories is an average of nearly 10 percentage points over-reported and for nearly 100 of the 150, the needle was touching over 15 percentage points adrift.  That means that many factories are missing an opportunity to gain 10 to 15 percent in productivity and reclaim perhaps a million dollars of NET PROFIT or hidden capacity that is not even being looked for any more.  Even more astounding, this same analysis found that most of the opportunity was available for recovery within six months by providing line-side staff with actionable information, empowering them, not just measuring them.  It is an idea being realized in many benchmarked companies who when they see what profit is available are connecting their factory workforce to make action unavoidable, instead of focusing on equipment monitoring alone. 

So before assuming all is well in the garden, and believing our current or planned “equipment-led MES approaches are, or indeed will make a difference, let’s remember the running shoes still un-scuffed in my travel bag.  I had my rose colored spectacles firmly on the bridge of my nose when I spoke the survey-taker in the street, but as leaders responsible for guiding the success of our businesses, we should be sure to remove those spectacles when we get to work and instead seek out those advisors that help us find the truth rather than what we want to hear. 

Taking comfort from flawed research with a positive spin is no protection from the harsh realities of a globalized economy and faltering consumer confidence.  In this time of “perfect storm” in the industry with rising commodity costs and pressure to keep retail prices squeezing profitability a sure-fire way to meet the challenge is to get realistic about it to begin with.  

The “Takeaway”
Benchmark your assumptions and your performance before making assumptions about the health of your factory and before assuming what medicine you should prescribe.  The answer is often refreshing; since when the truth is revealed it is rarely a surprise, and at least you can now do something about it.

Posted by Mark Sutcliffe on September 25, 2008 | Comments (0)