Sampling references - Consumer Goods
24 June 2010
The CAP Copy Advice Team regularly advises on advertisements where claims of product or brand superiority are based on data from consumer surveys. These types of claims are obviously entirely acceptable, but unless handled carefully they can face common pitfalls with the potential for an ASA investigation and an upheld adjudication.
Claims tend to fall into one of the following categories:
- Universal Claims; “People prefer product A”
- Sample Claims; “77% of women use product B”
- Preference Claims; “8 out of 10 cats prefer product C”
- Extrapolated Conclusions; “X% of people use product D”
- Inaccurate reporting of data; “the UK’s No.1 beer”
Universal claims can be problematic without robust data that is heavily in favour of a positive result i.e. statistically significant.
A complaint against Procter & Gamble Glamour/ Fairy Clean and Care (5th August 2009) challenged whether the claim “Glamour readers recommend Fairy” could be justified. Despite a footnote stating “Tested in a panel of 256 Glamour readers” the ASA upheld the complaint because it was considered that the results unfairly implied that 100% of Glamour readers would recommend Fairy Clean and Care when in fact they showed that 83% of the panel would do so. Because the results did not support the claim, it was judged to be misleading. The ASA also considered that the claim was misleading due to the confusing wording of some survey questions, with the format of some potentially predisposing respondents to circle an ‘agree’ rather than a ‘disagree’ option. Additionally, because a panel was used, respondents could have self-selected themselves and so may not have been representative of Glamour readers’ demography.
Two cases highlight the issues faced when actual survey results are used as a claim. The AA claimed that “98% of members that have used us would recommend us to a friend” (based on a 105 member sample). Similarly, a Procter & Gamble ad for Clairol claimed that “98% of Red magazine readers would recommend Nice ‘n’ Easy to a friend” (with support stating “in a survey of 245 Red magazine readers”).
The key issue for Nice ‘n’ Easy concerned the impartiality of responses. Those who completed the Nice ‘n’ Easy survey had previously registered for product trials. A sample product was sent to registered consumers who were also invited to enter a competition to win a trip to New York or to be entered into a prize draw for £50 John Lewis vouchers. The influence of the competition in particular was considered by the ASA to have potentially affected respondents’ impartiality, in order to stand a better chance of winning. In addition, because the survey link could have been forwarded to entrants’ friends, the ASA also considered that some may have falsely indicated they were Red readers to enter the prize draw. So whilst the results quoted may have been accurate, the survey design was considered to have undermined its validity.
In the case of the AA’s survey, complaints focused on whether the claim of the 98% recommendation was misleading because the sample (105 respondents) was thought to be too small to be representative of their 5.9 million members. However, because the sample was taken from members who had used the AA in the past two years with no evident methodological irregularities and because the ad had a super providing relevant details of the survey (i.e. number of people surveyed and not all members of the AA), this was seen as sufficient to qualify the claim. Additionally, the survey data was heavily in favour of a positive result.
In summary, the AA’s claim was substantiated with clear figures with no methodological influences, rather than a survey which was open to potential misreporting as was considered to be the case in the Procter & Gamble Clairol survey. Consequently, reporting survey results may not be enough to substantiate a claim on its own, with the survey methodology also a key factor to consider.
This raises the question about whether sample sizes should be placed in an ad, and whether that by adding them, they are sufficient to ‘fill in the gaps’ or to make the claim clearer. The Procter & Gamble Clairol and the AA cases illustrate the difficulties that can be experienced in this situation. In summary, the ‘small print’ or sample clarifier should be sufficient to support the statement without adding complexity, additional interpretive information or rendering the headline claim misleading. So it is not normally acceptable to make a claim and then attempt to qualify or clarify it via super or small print; these must support the statement or claim in full and not contradict it.
The CAP Code does not contain any explicit rules that sample sizes must be stated. However, the ASA tends to follow a general rule of thumb that if the sample size is not likely to be big enough to be robustly statistically significant to substantiate a headline claim, then qualifying text that gives the details, such as the sample size, should be stated in the advertisement. If results are based on a robust sample size, then it is not usually necessary to state the details. Because there are no hard and fast rules about what constitutes a small or large sample size, each advert is judged on a case-by-case basis.
Although CAP or the ASA do not provide advice on survey design methodologies such as sample sizes, questionnaire or discussion guide design, for example; we would encourage marketers to consider how studies are designed prior to them being conducted. If this has not been possible, for example when using secondary data sources; careful consideration and accurate reporting is advised.
The ASA would also consider whether any results are statistically significant in support of the claim. In the absence of statistical significance, it is unlikely that the ASA would accept the claim.
A complaint against Costa Coffee (16 June 2010) provides an interesting case. Costa Coffee claimed that “7 out of 10 coffee lovers prefer Costa” and “Starbucks drinkers prefer Costa”. The complaint was based on whether the latter claim could be substantiated because, as Starbucks pointed out, it was based on a different subset of survey results. There were two survey sets, one related to consumers who identified themselves as ‘coffee lovers’ and another set who identified themselves as ‘Starbucks drinkers’. While the claim “7 out of 10…” could be substantiated, the ASA sounded a note of caution because the headline and the qualifying claim were considered to be unclear and potentially confusing to consumers – one related to ‘coffee lovers’ and the qualifying claim related to a preference for Costa coffee by Starbucks loyalists. In addition, the small print related to the combined sample sizes, potentially adding further confusion. Nevertheless, the relevant aspects of the sample, when scrutinised individually by an external expert, were considered to have supported the claim and the complaint was not upheld. This points out that the ASA will take note of what the headline and qualifying statement will say in relation to each other and in relation to any small print. In this case, although the presentation was deemed confusing, the sample results did actually support the claims and the complaint was not upheld.
A Preference claim is one where the advertisement claims that a product is preferred over one or more other products. While a claim of this type can be true, again the survey questionnaire design can undermine the claim.
Flora Buttery (Unilever Ltd 25th February 2009) – made the claim, “more people prefer the taste of Flora Buttery” which was challenged as to whether the results were statistically significant. The complaint was upheld because the ASA considered that because the survey data showed that the number of people who chose a competitor brand or indicated ‘no preference’ was in fact higher than the number stating a preference for Flora Buttery, the result was not statistically significant to support a preference claim, and therefore misleading.
Unilever - Flora Pro Activ (31st October 2007), Colgate-Palmolive - Colgate toothpaste (17th January 2007) and Procter & Gamble - Iams pet food (26th March 2008). The advertisers claimed that “3 out of 4 doctors would recommend Flora…pro active mini drink”, “more than 80% of dentists recommend Colgate” and “8 out of 10 vets recommend Iams”, respectively. Once again, the basis for the upheld decisions was that the surveys were considered to have allowed multiple responses rather than giving single choice or ranked choice questions. Because the respondents could choose more than one product in the survey, the results would not be sufficient to make a clear preference claim
Claims based on extrapolated conclusions can also be problematic. eHarmony (18th November 2009) claimed that 2% of [American] newlyweds met on eHarmony. Unfortunately, for such a bold claim, the evidence was not compelling. The data was based on an online survey with the results being extrapolated to the whole American population who had married in that year, which the ASA viewed as an estimate rather than a robust interpretation of the data available. So in this case, because the claim was absolute, the use of extrapolated data to a whole population was not sufficient to substantiate the claim.
Inaccurate reporting of data
Inaccurate reporting of data was the basis of an upheld complaint against Wells & Young’s Brewing Company (12th November 2008) who claimed that their Banana Bread Beer and Waggle Dance were the “No 1 & No 2 flavoured premium bottled ales”. However, the data on which the claim was made showed that two other products (Hall & Woodhouse’s Golden Champion and Golden Glory) had higher volume and value sales compared to the Wells & Young’s products. Wells & Young’s had decided to interpret the data in a way in which consumers (in this case trade consumers) would not normally reference these products i.e. Banana Bread Beer and Waggle Dance were flavoured beers and competitor beers were ‘golden’ effectively dismissing their place in the rankings. The ASA determined that, because this was a message communicated to trade buyers and retailers, their interpretation would be that all four products could be recognised as flavoured beers and so the claim was considered to be inaccurate and therefore misleading, resulting in the complaint being upheld.
NB: while the public often have a reasonable amount of knowledge on which to base their complaint, it is often the case that when an ad is published in trade press, additional scrutiny is possible due to readers having detailed industry and market information.
Another case where a complaint was upheld due to reporting of data that was considered to be inaccurate was Beiersdorf UK (26th August 2009) who claimed that their DNAge Cell renewal day cream left ‘noticeably firmer looking skin’ with an asterisk to a claim that 126 women agreed. The advertiser did not state the overall sample size, which the ASA ruled could mislead. Additionally, it was not stated or made clear that among the 126 women who tested the product, only 70% agreed with the claim. Although the advertiser agreed that there had been an error in the footnote, the ASA nevertheless adjudicated that the ad was misleading and that claims could not be substantiated.
As ever, The Copy Advice team is best placed to consider individual ads and is on hand to provide free and timely bespoke guidance on any of the issues that are covered above: contact us on 020 7492 2100.