The Danger of Surveys

I was just leaving the Blockbuster Video store when I was
approached by a young lady with a clipboard.  She looked like
she had not been having much luck that morning, so, given I was not
in any sort of rush, I stopped to answer her questions. 

She asked about my exercise regime, calorie intake and alcohol
consumption.  As I considered my responses, I experienced a
twinge of guilt, as many of the questions centered around a healthy
lifestyle.  This youngster was clearly fit and healthy
herself, I didn’t want to disappoint myself or her. 

So, let’s say I was not as truthful as I might have
been.  I considered the last week, when I had travelled to
Chicago and Boston and fine-dined on both occasions; not to mention
the wine.  I rationalized to myself that this was not a
“normal” week, and as such I should regard these events
as “outliers” to be excluded from my
self-assessment.  The fact that I had carried my running shoes
and shorts in my luggage, but somehow had not found the time to
dust them off for an outing, was surely out of character. 

This is not exactly a deception – I would rather phrase it as
a “softening” of the harsher truths that I feel
righteously entitled to – since I had given up my time to the
surveyor for free.  What’s more, I thought, since the
sample size was probably enormous, my answers and their accuracy
would not skew the outcome.  My conscience was clear.

On reflection, I later managed to rationalize a viewpoint from
which my answers have the ring of truth to what I presume to be the
intent of the survey.  How so?  Because my answers
reflected the “me” I fully intend to be in three or six
months’ time; in essence, my future state as I imagine it to
be.  Much like a quiz, I believe I have calculated what the
right answer “ought to be.”  Surely this is what
the researcher was getting at? 

And What About the Truth?
I would be being truly unkind on myself to step back and observe
that this future state has been three to six months away, for many
years…perhaps even a decade.  Ouch.  Just as well
the survey was not an important piece of research. 

And there lies the problem with surveys.  We all know that our
answers are a reflection of ourselves, and it is in our nature to
“put our best foot forward.”  But our “best
foot” is not often the truth.

And Why Does It Matter?
I reflected on this story when I read a survey of food makers
stating that the Overall Equipment Effectiveness (OEE) in most food
plants today is 88 percent.  This is completely
unbelievable—period.  I know from our own benchmarking
work with well over a 150 plants that this cannot be true. 
Indeed, I assert that it is not only unbelievable, it is
dangerous.

The phenomenon I described in my story is aligned with the OEE
statistic in the report, regularly adjusted from plant to plant to
reflect “local” conditions and to be “just a
little more forgiving” (some with “breaks”, some
without, some with certain changeovers, some without, some with
scheduled downtime and some without… some collecting most
downtime incidents, some not, you name it).  Certainly, these
food producers were reporting their desired future state, not their
true current state.

It’s Not Life And Death, This Is Much More
Serious 
There is—and should be—a growing concern about the
difference in perspective between survey-taker and
survey-respondent.  The survey taker reports her findings with
all seriousness—she is looking for meaningful trends on which
to report and to influence action.  Yet, the survey respondent
answers the survey hopefully and even trivially and his answers are
completely unverifiable.  This is not a good basis for policy
making in business .  I’m sure you’ve heard the
apocryphal quote “This is about money, so come-on, it’s
not life and death…this is much more
serious!” 

This phenomenon is dangerous because if we believe that optimistic
survey data is true, we eliminate our belief and motivation to
focus on and act on significant improvement.  For example, if
I believe that I am already at my desired future physical
condition, why bother to dust off my running shoes!

 An OEE of 88 percent, if it were true, would leave little, if
any, real need or room to improve.  Yet, from recent
benchmarks of over 150 locations that I have been reviewing, the
gap between “reported” and “actual” output
performance of U.S. and UK food factories is an average of nearly
10 percentage points over-reported and for nearly 100 of the 150,
the needle was touching over 15 percentage points adrift. 
That means that many factories are missing an opportunity to gain
10 to 15 percent in productivity and reclaim perhaps a million
dollars of NET PROFIT or hidden capacity that is not even being
looked for any more.  Even more astounding, this same analysis
found that most of the opportunity was available for recovery
within six months by providing line-side staff with actionable
information, empowering them, not just measuring them.  It is
an idea being realized in many benchmarked companies who when they
see what profit is available are connecting their factory workforce
to make action unavoidable, instead of focusing on equipment
monitoring alone. 

So before assuming all is well in the garden, and believing our
current or planned “equipment-led MES approaches are, or
indeed will make a difference, let’s remember the running
shoes still un-scuffed in my travel bag.  I had my rose
colored spectacles firmly on the bridge of my nose when I spoke the
survey-taker in the street, but as leaders responsible for guiding
the success of our businesses, we should be sure to remove those
spectacles when we get to work and instead seek out those advisors
that help us find the truth rather than what we want to
hear. 

Taking comfort from flawed research with a positive spin is no
protection from the harsh realities of a globalized economy and
faltering consumer confidence.  In this time of “perfect
storm” in the industry with rising commodity costs and
pressure to keep retail prices squeezing profitability a sure-fire
way to meet the challenge is to get realistic about it to begin
with.  

The “Takeaway”
Benchmark your assumptions and your performance before making
assumptions about the health of your factory and before assuming
what medicine you should prescribe.  The answer is often
refreshing; since when the truth is revealed it is rarely a
surprise, and at least you can now do something about it.