Don’t Always Trust Customer Surveys

- - Working

I post a lot of quotes from interviews with your customers (PE’s, AB’s, AD’s) here and I wanted to point out something that’s rarely mentioned. Listen to what people say, but trust your gut and your numbers and always experiment to find things that work for you. Everyone (except experts in the survey field) asks questions that lead the interviewee to the answer they wanted in the first place and people generally give answers that are idealized. They say photographers should do X, Y and Z, but then ignore those directives when in an actual hiring situation. Here’s a great example of what I’m talking about:

You’re with Walmart. It’s 2009, and you want to do something new, something transformative, to out-innovate rival Target. You have a sense that Target is cleaner, better designed, less cluttered. Walmart aisles are crammed, packed, an infinite jumble of product.

So you’re thinking of launching an uncluttering project. Strategic. Huge. Millions of dollars. But before you make any changes, you want to float the idea by customers.

So you conduct a survey, asking customers: would you like Walmart aisles to be less cluttered? And they say, “Yes, now that you ask, yes, that would be nice.” And you check the box by “customer input” and report back, hey everyone, good news, yes, customers like the idea.

Walmart spends hundreds of millions of dollars uncluttering their stores, removing 15% of inventory, shortening shelves, clearing aisles. Yes, it’s expensive and time-consuming, but this is what customers said they wanted, so you barrel through it.

You’ll never guess what happens now. (Actually, you’ve probably already guessed, but it sounded better to say you’ll never guess.)

Sales went down. Way down. I mean waaaaaay down. I’m talking, from the beginning of that project until today, Walmart has lost over a billion dollars in sales. (Yes, billion with a “b”.) It’s actually closer to two billion dollars of sales they missed out on, and maybe more.

Needless to say, the executives in charge of the project have been fired, and Walmart is spending yet more money to return to its original, time-tested strategy of offering a huge (albeit cluttered) inventory at low prices.

Read it here.

What people say isn’t always the same as what they do.

There Are 15 Comments On This Article.

  1. This is my attitude when I read all those “I hate email promos” rants from AD’s. I figure they probably do hate them. But they also look at them, and respond to some of them.
    Ergo….I do email promos. I can’t afford to lose a billion dollars in sales.

  2. So rather than be better, cater to the lowest common denominator? I know that’s not your message with this post… But the walmart story basically says”why make things better, people love crap.”

  3. I once worked at a pretty smart ad agency, we sometimes did focus groups.

    There were strong unwritten rules there about how focus groups were NOT allowed to see and give feedback on creative work.

    Focus groups were used before any campaign started, to try to figure out their hopes and fears and whatever central insights we can get into their gut-level thinking. This was all about how we can find the target market’s real motivations, not by asking direct questions, but by asking lots of indirect questions and gently watching what happens….

    We never asked them what they thought about any advertising. Ever.

  4. The message is simple and one that I came to realize a few months ago. With all of these so-called, marketing strategies for photographers, it boils down to the simple analogy described here. Basically, what works for Target, doesn’t work for Wal-Mart, because they’re two different stores, with some variations in their customer base. I would dare to say that even with their shared customer base, some of those customers go to the two different stores for different reasons. They don’t expect the stores to be the same.

    Lesson I’ve learned from all the info I’ve gathered about marketing in the new media age? Do what I do and do it as well as I can and target MY audience. Following shooting trends and marketing strategies of other photographers doesn’t work, unless it fits what I do and my audience.

    My 2 cents worth…..

  5. Jeb Tilly

    Jeff Callen seems to have it right. As the saying goes, companies tend to use research like a drunk uses a streetlight: for support rather than illumination. To me the lesson isn’t that we shouldn’t trust research, but that we should understand how to use it. Good exploratory research was fundamental to making millions for Domino’s Pizza recently. That research never asked people what Domino’s should do, but rather explored how they felt about the brand and its products. Worth noting as well that Wal-Mart’s sales decline is not the effect of one bit of bad research alone. It has to do with a broader shift in brand strategy that alienated core customers.

  6. Wal-Mart was also advertising in Vogue for a while … as if. I wonder if it was the same marketing group who made that insightful (sarcasm) media buy. BTW – that was prior to their move to Martin.

  7. Sameol Story

    Most business data are used to support decisions that have already been made rather than guide decision making. The Wal-Mart story isn’t really surprising — it happens every day.

    But trusting business reporting is also dangerous. It’s actually pretty difficult to lose $1B. I’m guessing Wal-Mart’s losses resulted from a multitude of issues — some self impact (e.g., bad decisions and unpopular business practices) and some external factors (e.g., economic conditions, changing customer buying habits, etc.). If you wanted to understand what really happened, you’d have to spend a fair amount of time analyzing it.

    Big companies don’t like analyzing their mistakes. That would ultimately result in assigning blame — no one wants that. It’s much easier to focus on developing a new plan to solve the problems created by the last plan. That way the usual middle-management suspects can all compete for the right to lead And, of course, some data would help sell your proposal — maybe a nice customer survey would do the trick. If you’ve ever wondered by big companies so rarely do anything innovative, just work for one for a few years.

  8. I read an article about the Walmart decluttering, and the gist seemed to be that when a store is cluttered with stock piled high everywhere there is a perception that the store is cheaper than one which is tidy and neat. So the customers weren’t lying when they said they would prefer the store to be less cluttered, but the customer survey did not understand why people were shopping there.

    One could equally take down all outdoor advertising in response to a survey showing that people don’t like it, but when sales fall this wouldn’t mean that the people were wrong to say they didn’t like the outdoor advertising.


  9. Should South Africa allow Walmart into South Africa ?
    The South African government is worried that it might actually cost jobs , i see New York has restricted them in some areas.
    I wonder what a focus group would feel ?

  10. I think this is the most useful post I’ve read of Rob’s (and I’m a big fan of this blog).

    And back to the core of what I think he was saying:

    Trust your instinct, and continue to try things out, regardless of consensus. You read the surveys to get an idea of what’s on some peoples’ minds, understanding that you only need to connect with 20 clients in a year, and that not everyone has the same needs and buying habits as the person being surveyed.

    It’s also difficult for people to truly know why they hire you (as a consumer yourself, for example, do you know why you just bought a Coke instead of a Pepsi?)

    Yes, they may have seen your work on Foundfolios and called you up. But did they recall that you’d been sending them email promos for the past 6 years, print promos for the last ten, called them on occasion to arrange portfolio meetings, and stalked them on Linkedin? Did those communications subtly embed themselves in their subconscious over the years, so that your name sounds familiar when they see you in a magazine photo credit?

    And did they just answer one of those surveys that this blog post alludes to, and say that they NEVER want to receive another email promo? Is that because 1,000 photographers sent them the wrong images at the wrong time, and you’re the 1,001th email promo which happens to include an appropriate photograph that fits an upcoming project?

    Also, did the images that you showed them change in style, content, or sophistication over the years? And did they never have a need for your specialty of photography, until the one day that they scored a new client who did? Or did the company get a new art buyer or creative director who now connects with your work? Or did your website edit just get a lot more interesting or appropriate to them?

    You won’t know if you stop keeping in touch, sharing your work in a variety of ways, and expanding your portfolio in ways you never imagined.

    And neither will they.

  11. T. C. Knight

    Not really a thorough example because what isn’t said is that WalMart decluttered alright, but only left WalMart brand products on the grocer shelf as well as in other parts of the store. My wife does not go to WalMart to buy generic, but rather name brand at better prices. When THAT dissappeared she switched to the grocery store and left WalMart behind.

    Hhhmmm…so, since surveys say photobuyers have no budget for photography, we have, to reduce the costs, begun giving buyers generic.

    Is there a parallel here?

  12. Couldn’t they have tried this at ONE or TWO stores and measured the results before going whole hog and spending bazillions of dollars changing many stores? Duh.