Survey Says: Polls Can Be Wildly Misleading

You should include more surveys in your writing. But be careful. Here are six red flags indicating a survey is bogus.

I’m conducting a survey. So here’s a question for you. Do you think surveys are biased?

I won’t be reporting the results, because they will be meaningless. You’ll soon understand why (it has to do with acquiescence bias and lousy sampling methodology, at least). Meanwhile, here’s my two-part message for journalists and other writers of mainstream nonfiction:

1. When a properly conducted survey or poll adds helpful perspective to a topic, including the results can make your story more compelling and useful. In my experience as an editor and publisher, surveys are significantly underused by journalists and other writers (except at election time, when suddenly they’re everywhere).

2. However, writers need to be extremely careful, because surveys are often really poorly done. Even the best ones represent only a slice of the population at a point in time, and therefore an approximation of what everyone really thinks. So just as with reporting statistics that can be wildly misleading, writers need to understand the difference between credible and incredible surveys.

Note that the terms poll and survey are often used interchangeably by the people who conduct them, though survey tends to imply something more in-depth. For this post I make no distinction.

Here’s the takeaway: Many surveys are woefully inaccurate and downright misleading. That’s probably true for most of them, given the sheer volume of poorly done, often self-serving marketing surveys that have no business showing up in credible writing. Below you’ll find six red flags to look for. First…

What makes a credible survey?

Finding surveys is not difficult. Simply search the internet for “my topic, poll, survey.” Finding credible surveys is the tricky part. Here’s what to look for: