In the first panel, statistics were presented about the use and acceptance of consumer perception surveys before NAD. Interestingly, consumer surveys are not used as much as one might expect in cases before NAD. Over the last five years, only 36 cases used one or more consumer perception surveys (about 8% of cases). Out of 52 surveys that were submitted, 42 were in support of the challenger and 10 were in support of the advertiser. Thus, given the low number of surveys that are presented to NAD, just submitting a survey could help to distinguish a company appearing before NAD, especially if your ad is the one being challenged.
However, if an advertiser is going to submit a survey, then it is important to work with counsel and an expert to get the survey right. Surprisingly, 62% of consumer perception surveys submitted to NAD were rejected for having no value. So where are advertisers going wrong? The most common reasons the surveys were rejected were because the survey used leading closed questions, there were issues with the control questions, an improper control stimulus, questionable coding of open-ended questions and inadequate or improper filtering. Other reasons were reliance on closed questions (open-end questions absent or ignored), biased presentation of test stimulus, wrong population, no control group or condition, the survey tested consumer beliefs rather than their perceptions of the ad, and there was no “don’t know” option.
In two subsequent panels, survey experts and NAD attorneys gave important guidance during the conference on what makes a good survey. NAD noted that it likes when the experts participate in NAD meeting to explain the survey and to be able to respond to critiques about the survey. NAD also noted that lawyers should not just give NAD a summary of the survey, instead NAD wants to see the actual report and data to aid in their decision.
The survey experts explained that the generally accepted survey structure contains (1) an appropriate sample in composition and size drawn from the population of interest; (2) a valid design with exactly matched cells, realistic presentation of stimulus, controls to account for bias and noise, and progression from open-ended to non-leading closed ended questions; (3) a questionnaire with unambiguous and unbiased questions and the use of appropriate filters for focused open-ended or closed-ended questions; and (4) competent statistical analyses.
NAD attorneys gave guidance on what they believe a reliable survey includes and listed the following factors: (1) an appropriate universe and representative sample; (2) a well-designed questionnaire; (3) clear and non-leading questions; (4) use of a proper control cell; (5) use of appropriate survey stimulus; (6) data analyzed with accepted practices; and (7) reliable survey methods and procedures. NAD attorneys noted that as a general rule, NAD has been persuaded by arguments that open-ended questions are better indicators of how consumers interpret a commercial message because respondents answers are not controlled by the suggestions contained in the questions themselves. They recommended that the survey begin with open-ended questions and filter down to more close-ended questions.
During the question and answer panel, some interesting topics were discussed. Among them, was the impact of technological advances on surveys. For instance, consumers (especially millennials) are using their mobile phones more often these days and it is often difficult for survey experts to get young consumers to respond to surveys in a timely manner when the survey needs to be done on a computer. NAD indicated that if there is evidence that consumers are viewing the advertisements on their mobile phones, then they might accept surveys that are done on mobile phones. NAD indicated that if survey experts deviate from the norm, then they need to have a good reason and NAD wants to know the reason. While there are potential issues with consumers taking the survey on their phones, such as how to deal with open-ended questions and how to ensure that consumers cannot move to the next question prior to answering as is the case with current controls, NAD will follow the standards set by the industry.
TAKEAWAY: Consumer perception surveys can be very beneficial in NAD cases when done correctly. Thus, when bringing or defending a case before NAD it is important for companies to consider whether a consumer perception survey is advisable and if so, for companies to work closely with their legal counsel and expert so that the survey is properly tailored to meet NAD’s standards.