Copywriters are Not Always Scientists

This week’s Participatory News assignment is to fact-check a dubious claim found out in the wild.

I enjoy skimming Men’s Health magazine each month. It combines useful nutritional information, workout plans, and the occasional life advice about topics like careers and personal finance. More than anything, the ever-present photos of salmon and blueberries remind me to buy and eat such foods.

The publication is guilty, however, of a classic media tactic: They publish an incredibly short summary of a recent study, and then tack on their own advice on how to work the findings into your life. To be fair, I’ve seen many newspapers and magazines use this formula. It’s the natural result of cramming science journalism into 2 sentence blurbs. But there are a few big problems with this format.

First, a single study is rarely, if ever, enough evidence to warrant a change in behavior. Anyone who’s ever read surprising results in a study and then outright laughed upon getting to the Methodology section understands why. Many studies are conducted on twenty graduate students, or actual lab rats, or the design of the study is clearly biased towards the eventual results. I’m not the kind of guy who reads studies for fun, but these problems are fairly obvious when you read the original paper. It’s one reason scientists wait until there is clear agreement across a variety of research before advising action. It’s also why they always seem to conclude a study with the line “Further studies are needed” (besides the fact that “further funding is needed”). As one example, the very study we’ll be looking at substantiates its claims regarding what men and women seek in a mate only after finding consensus among “studies that have spanned 20 years and often include international datasets with sample sizes in the tens of thousands.” When a magazine summarizes a single study in a single sentence, we aren’t provided nearly enough context to think critically about how the results were achieved. We’re just told to believe them.

Even lazy science journalism serves a purpose. The publishers take a dense academic study and make it accessible for a mass audience. And perhaps by interpreting the results into actionable advice, they believe they’re helping the lay reader. What they’re often doing, however, is making unsubstantiated claims on the back of a scientific study (itself of varying quality and methodology). By including their own advice in the same small paragraph as the peer-reviewed study, the publication encourages its reader to extend the scientific study’s credibility to the copywriter’s tidbit.

This is all fine and good when we’re talking about new strategies for bigger biceps. The advice regarding how to interact with women, however, borders between wildly misogynist and downright hilarious. We could probably solve overpopulation if every man did what Men’s Health says to do, and every woman behaved as Cosmo advises. I’ve known how silly these prescriptions were since, oh, puberty, but I thought I would take the opportunity of this week’s fact-checking assignment to look into some of the advice and see what the studies referenced actually found.

The magazine has a regular feature dedicated to relations with women, titled SEX BULLETIN (to be fair, there are also more thoughtful, longer pieces about women). Five findings are presented on one page:

Men's Health Sex Bulletin

  1. “The Last Longer Jab”…”An injection of hyaluronic acid gel in the head of your penis” may delay premature ejaculation “by about 6 minutes.”
  2. “Women fake orgasm to keep men from straying”…”That phony “O” may be her attempt to show she’s committed and reduce the chance that you’ll cheat”…”Other ways she reels you in: flirting with guys in front of you and calling at unexpected times.”
  3. Three percent of women say using lube makes them feel inadequate.
  4. “Women are less likely to regret random hookups if the sex was satisfying.” (This advice may be provided to obscure the well-documented research establishing that in general, women are more sociosexually restricted, or less desirous of casual sex than men (Schmitt, David P., 2005)).
  5. And lastly, the one fact I might reasonably be able to check:

Wanted: Self-Made Millionaire

Ninety percent of women prefer a long-term partner who earned his money rather than inherited it. “Women associate self-earned wealth with reliability, self-sufficiency, intelligence,” says study author Peter Jonason, Ph.D. If you have family money, play up your generosity– say, donate to charity. That shows you don’t take wealth for granted. (emphasis mine)

The study referenced provides a quote from its author, and no other identifying information. We’re lucky. In other examples, the factoid in question is followed by vague statements like “according to Japanese research.”

Taking a closer look at the study referenced, the results come from a sample size of 145 women. More importantly, the abstract concludes that “In sum, financial security appears to have minimal effects and associations on mating psychology despite the paramount role that sociocultural psychologists argue it has” (Jonason, ix). So, the study’s author is actually arguing the exact opposite of Men’s Health selective quotations: Financial security was found to have less of a role than has traditionally been argued.

I wasn’t able to find any scholarly work regarding Men’s Health‘s advice that men should make up for inherited wealth (and a lack of reliability, self-sufficiency, and intelligence) by donating to charity and then telling women about it. Perhaps we can design such a study. Just don’t ask to see the methodology.

This entry was posted in All, Fact-checking Assignments and tagged by mstem. Bookmark the permalink.

About mstem

Matt's a Research Assistant at the Center for Civic Media at MIT's Media Lab. He has spent his career at the intersection of technology and social change. He graduated with high honors from the University of Maryland College Park, where he wrote a thesis on the disruptive role of political blogs in journalism. He went on to join the strategy team at EchoDitto, a boutique consulting firm building cool technology for nonprofits, startups, and socially responsible businesses. Matt went on to direct new media at Americans for Campaign Reform, a bi-partisan grassroots effort, and the New Organizing Institute, where he helped to train the next generation of organizers. For most of this time, he also ran one of the most popular NetSquared groups in the world.

6 thoughts on “Copywriters are Not Always Scientists

  1. The fact you checked is interesting to me because it’s a great example of a self-affirming fact. It’s not written to hereditary millionaires, who are presumably reading something else. It’s written to the rest of us who will congratulate ourselves for being hard-working earners who don’t donate to charity.

  2. Matt, I really appreciated your analysis of the shortcomings of some scientific journalism. In discussion in class, you made one of your main points more forcefully – if you can’t find the original study being cited, it’s virtually impossible to see if it’s being presented correctly.

    It might be interesting to see what could be done to make these short pieces more helpful or fairer. One is obviously linking to the original studies. Another might be providing a quick summary paragraph – why you should care, why you shouldn’t take this too seriously. Another might be a simple mouse-over that listed how many data points are in the study, or gave a quick way of looking at the “p” value of a study, the chance that the study’s results could also be explained by sheer randomness…

  3. Pingback: Study Grader | News and Participatory Media

  4. Pingback: Study Grader

  5. This is really exciting! I don’t think you’ve taken into account what happens day to day, but still a good post.

Comments are closed.