Part Four – Online Surveys Must Cast a Wider Net and Pay Incentives
When we look outside the radio industry for other businesses using online consumer surveys to guide strategy, there is no shortage of confidence in the online survey approach. Wherever we looked - Food, Cosmetics, Travel, Clothing, Pharmaceuticals, Political – it seems everyone else is doing online research and is supremely confident in the results. So what are they doing that Radio is not? Two answers stand out. Broader panel recruitment and paid incentives.
In the past few weeks we have looked at many examples of online research conducted for other industries targeting consumers and professionals. Several incorporate intricate multi-media in their surveys. A large supermarket chain uses virtual store shelves and a shopping cart. Coors Light showed a storyboard for a potential TV spot. Levi’s used photo images to test preferences for pocket designs and preferred fading of blue jean colors. Southwest Airlines tracked brand association to different traveler benefits. GlaxoSmithKline probed doctors on heart disease. If you think it’s hard to find radio listeners to do a music test, imagine how hard is it to get high paid, busy cardiologists to sit through an hour long pharmaceutical survey. But these surveys are all done online and the confidence level in the results at these major corporations is extremely high.
In a nationwide telephone callout survey conducted by Kelly Music Research, of 362 music radio P1’s aged 18 to 54, only 9% of callout respondents report being regular participants in online Radio surveys. However, 46% report participating in online surveys for other industries. When we asked our telephone callout panel if they would participate in online Radio surveys that pay an incentive, the number who said “Yes” jumped to 72%.
To get more qualified respondents and reliable data from online research, we need to cast a wider net. Broader recruitment strategies for online surveys, such as telephone outreach are necessary. Incentivizing panelists with a paid premium will further increase online research response rates. Unfortunately, offering “a prize” that a participant “might win” is not enough incentive for most people.
As radio stations populated their e-mail databases, many hoped it would lead to a cheap way of getting reliable research. Cheap yes, reliable no. Most radio station databases were built around contests and promotions. As mentioned earlier in our study, radio station databases attract a disproportionate number of Extreme Fans and Contest Players. Many in station databases are not in the defined target age groups and don’t even listen to the station.
Our study shows that a very small percentage of listeners in station databases clear the screening criteria for a standard radio music survey. We screened several client stations’ listener databases for potential inclusion in a music test. On average, about 1% of database members made it through a standard music test screener. These website database members were offered a cash premium to participate in an independent online music survey. No prizes or sweepstakes were offered. Overall, about 3% of all database members expressed interest. However, when screened for age, gender, ethnicity, and music preference, the number dropped to 1%. And after filtering the list for client station P1’s and P2’s, the qualified percentage dropped even lower.
In major markets and small markets, we’ve seen stations offer database members big prizes and substantial incentives to participate in music surveys. Some of these have attracted several hundred respondents. However when the stations drilled the data down to their actual target demographic, the sample became too small to be reliable.
We believe music testing can include radio station database members, but not rely on them exclusively. Taking a cue from other industries, online Radio research needs to be more inclusive by casting a wider net. And we need to reward participants for sharing their honest opinions with us.
Thursday, January 28, 2010
Tuesday, January 12, 2010
The Differences between Callout and Web database Music Surveys
Part Three – The Prize Pig Factor
As we study the factors that lead to different music research results generated by random Callout samples and online station web database samples, we are taking a closer look at the listeners who make up both types of panels. So far, we have seen that Callout sample panels attract participation by more Passive listeners or Normal Fans. Station website databases are more likely to be populated by the vocal minority Actives or Extreme Fans. In our “research of the research” we found another group that is capable of corrupting research drawn from surveys using radio station databases – “Prize Pigs.” It is quite likely that Prize Pigs are in your station’s database more than once. A Prize Pig is probably not even a listener of your station except for the contests. And it is likely that the Prize Pigs in your database are also in the database of every other station in the market. Radio stations using contests as a point of entry and motivation to build the station’s e-mail database should beware. Stations relying on that same database for market research should reconsider. And particularly risky is mixing both – offering the chance to win a very cool prize for participating in a survey.
Our nationwide Callout sample of over 300 music radio station P1 listeners aged 18 to 54 in the top 25 markets reveals that only 8% of listeners have “High Interest” in radio station contests. The majority of listeners we reached in our Callout survey don’t participate in contests because they believe they’d never win. However, Prize Pigs are drawn to the challenge and the opportunity to get something for nothing.
There is a story we were told by a Radio consultant about contests. Around 1980 a Los Angeles radio station gave away a Ferrari sports car in a major on air contest. One lucky listener would be randomly drawn. Spreading the promotion over several months, hundreds of thousands of listeners fantasized about winning this very expensive performance car. Finally, a winner was selected – a very lucky and excited guy from L.A. The next year, the radio station decided to do the exact same promotion a second time. Another Ferrari. Same contest. Same buildup. Same excitement. And then the winner was drawn – same guy. The same guy had won two Ferraris in two consecutive contests. How could this happen? It turns out he was a full time contest player. He knew how to stack the deck in his favor. Today, there are many more like him.
They call themselves Contesters, Gamers or Players. Radio uses less flattering names - Prize Pigs or worse. They try to win anything your station is giving away. And our examination of Callout and Station Web-based panels suggests that Prize Pigs may be polluting radio station databases across the country. In fact this small but aggressive contest cult is more pervasive than we knew. In a recent Google search for “secrets to winning radio station contests,” more than 480,000 results surface. There are Contest Clubs - people who compete with each other tallying winnings. There are tips and suggestions for increasing the odds of winning. There are stories of people who play contests as a full time source of income. Kathy, one Contester reports winning at least twice a month – merchandise, tickets, vacations – she hasn’t paid for a trip in years. Winnings that Prize Pigs don’t use themselves, they can sell on e-Bay or StubHub for cash. Here is one example of a website that offers tips to Prize Pigs who want to win radio contests: http://contests.about.com/od/improveyourodds/tp/radiocontests.htm
Another site offers this piece of advise for increasing your chances of winning station database radio giveaways: “Submit the maximum that you can. If there is no maximum, flood the station with your entries.” If your station’s e-mail database is an entry point for giveaways, chances are you’ve been flooded. Between AOL, Gmail, Yahoo, Hotmail and all the others, one individual has unlimited ability to create as many e-mail addresses he or she would like, then seed your database with all their aliases.
One Contester claims to be in radio station databases with over 100 different e-mail addresses. This greatly improves his chance of winning e-mail entry contests. And if the station does an on-line music survey invitation with a chance to win an Ipod or something even better, this one Prize Pig will get 100 survey invitations and complete 100 different music surveys for 100 chances to win. And he may not even cume the station. Since these Prize Pigs may not even listen to your station or format, this can explain survey results that don’t look right. Radio stations report to us that when they do on line surveys to station databases with a hot prize incentive, they get many more completed surveys. “The bigger the prize, the larger the sample,” says one PD. But is it more people? Or is it just a few Prize Pigs completing multiple surveys?
When Prize Pigs who are not listeners participate in on-line music surveys, they will try to make themselves look legitimate. As the Contester websites suggest, radio stations want to award prizes to listeners. So Prize Pigs pretend to be real listeners. Even if they don’t recognize a single song in a survey, the Gamer will still score the songs familiar. They typically give higher scores to songs by artists whose names they recognize. In station database surveys which display artist names, it is common to see brand new releases by established, big name artists score very familiar and very positive after getting only very little radio airplay. Prize Pigs trying to pass themselves off as real listeners will also give lower scores to songs by artists who’ve gotten negative press or may have lost the cool, hip image. Recent examples of brand new releases from artists with highly recognizable names include AC/DC and Miley Cyrus. While unaided Callout was showing typically low Familiarity scores for their new songs, station database online tests with artist names on display showed instant Familiarity for both. In terms of Appeal scores, station database samples scored the new AC/DC song overwhelmingly positive and the new Miley Cyrus song overwhelmingly negative. In both of these cases, the respondents seemed to be scoring the artist name, regardless of what the song sounded like. Results like these can be indicative of non-listeners participating in on-line database surveys.
It’s important to remember that our findings do NOT suggest that Radio contests have negative appeal. In fact, 79% of our sample report having at least some interest in contests, but they are not Players. They enjoy the entertainment value of trivia questions, radio games and contests in the same way TV game show viewers enjoy watching the contest and playing along in their heads.
Critical to the success of the research process is to control sampling on the front end to ensure the integrity of the data on the back end. Panels built on individuals who opt-in to the database for reasons such as contesting relinquishes control and weakens the statistical reliability of the sample and ultimately the final data.
Next time, online surveys that Normal Radio Fans will do.
As we study the factors that lead to different music research results generated by random Callout samples and online station web database samples, we are taking a closer look at the listeners who make up both types of panels. So far, we have seen that Callout sample panels attract participation by more Passive listeners or Normal Fans. Station website databases are more likely to be populated by the vocal minority Actives or Extreme Fans. In our “research of the research” we found another group that is capable of corrupting research drawn from surveys using radio station databases – “Prize Pigs.” It is quite likely that Prize Pigs are in your station’s database more than once. A Prize Pig is probably not even a listener of your station except for the contests. And it is likely that the Prize Pigs in your database are also in the database of every other station in the market. Radio stations using contests as a point of entry and motivation to build the station’s e-mail database should beware. Stations relying on that same database for market research should reconsider. And particularly risky is mixing both – offering the chance to win a very cool prize for participating in a survey.
Our nationwide Callout sample of over 300 music radio station P1 listeners aged 18 to 54 in the top 25 markets reveals that only 8% of listeners have “High Interest” in radio station contests. The majority of listeners we reached in our Callout survey don’t participate in contests because they believe they’d never win. However, Prize Pigs are drawn to the challenge and the opportunity to get something for nothing.
There is a story we were told by a Radio consultant about contests. Around 1980 a Los Angeles radio station gave away a Ferrari sports car in a major on air contest. One lucky listener would be randomly drawn. Spreading the promotion over several months, hundreds of thousands of listeners fantasized about winning this very expensive performance car. Finally, a winner was selected – a very lucky and excited guy from L.A. The next year, the radio station decided to do the exact same promotion a second time. Another Ferrari. Same contest. Same buildup. Same excitement. And then the winner was drawn – same guy. The same guy had won two Ferraris in two consecutive contests. How could this happen? It turns out he was a full time contest player. He knew how to stack the deck in his favor. Today, there are many more like him.
They call themselves Contesters, Gamers or Players. Radio uses less flattering names - Prize Pigs or worse. They try to win anything your station is giving away. And our examination of Callout and Station Web-based panels suggests that Prize Pigs may be polluting radio station databases across the country. In fact this small but aggressive contest cult is more pervasive than we knew. In a recent Google search for “secrets to winning radio station contests,” more than 480,000 results surface. There are Contest Clubs - people who compete with each other tallying winnings. There are tips and suggestions for increasing the odds of winning. There are stories of people who play contests as a full time source of income. Kathy, one Contester reports winning at least twice a month – merchandise, tickets, vacations – she hasn’t paid for a trip in years. Winnings that Prize Pigs don’t use themselves, they can sell on e-Bay or StubHub for cash. Here is one example of a website that offers tips to Prize Pigs who want to win radio contests: http://contests.about.com/od/improveyourodds/tp/radiocontests.htm
Another site offers this piece of advise for increasing your chances of winning station database radio giveaways: “Submit the maximum that you can. If there is no maximum, flood the station with your entries.” If your station’s e-mail database is an entry point for giveaways, chances are you’ve been flooded. Between AOL, Gmail, Yahoo, Hotmail and all the others, one individual has unlimited ability to create as many e-mail addresses he or she would like, then seed your database with all their aliases.
One Contester claims to be in radio station databases with over 100 different e-mail addresses. This greatly improves his chance of winning e-mail entry contests. And if the station does an on-line music survey invitation with a chance to win an Ipod or something even better, this one Prize Pig will get 100 survey invitations and complete 100 different music surveys for 100 chances to win. And he may not even cume the station. Since these Prize Pigs may not even listen to your station or format, this can explain survey results that don’t look right. Radio stations report to us that when they do on line surveys to station databases with a hot prize incentive, they get many more completed surveys. “The bigger the prize, the larger the sample,” says one PD. But is it more people? Or is it just a few Prize Pigs completing multiple surveys?
When Prize Pigs who are not listeners participate in on-line music surveys, they will try to make themselves look legitimate. As the Contester websites suggest, radio stations want to award prizes to listeners. So Prize Pigs pretend to be real listeners. Even if they don’t recognize a single song in a survey, the Gamer will still score the songs familiar. They typically give higher scores to songs by artists whose names they recognize. In station database surveys which display artist names, it is common to see brand new releases by established, big name artists score very familiar and very positive after getting only very little radio airplay. Prize Pigs trying to pass themselves off as real listeners will also give lower scores to songs by artists who’ve gotten negative press or may have lost the cool, hip image. Recent examples of brand new releases from artists with highly recognizable names include AC/DC and Miley Cyrus. While unaided Callout was showing typically low Familiarity scores for their new songs, station database online tests with artist names on display showed instant Familiarity for both. In terms of Appeal scores, station database samples scored the new AC/DC song overwhelmingly positive and the new Miley Cyrus song overwhelmingly negative. In both of these cases, the respondents seemed to be scoring the artist name, regardless of what the song sounded like. Results like these can be indicative of non-listeners participating in on-line database surveys.
It’s important to remember that our findings do NOT suggest that Radio contests have negative appeal. In fact, 79% of our sample report having at least some interest in contests, but they are not Players. They enjoy the entertainment value of trivia questions, radio games and contests in the same way TV game show viewers enjoy watching the contest and playing along in their heads.
Critical to the success of the research process is to control sampling on the front end to ensure the integrity of the data on the back end. Panels built on individuals who opt-in to the database for reasons such as contesting relinquishes control and weakens the statistical reliability of the sample and ultimately the final data.
Next time, online surveys that Normal Radio Fans will do.
Monday, January 4, 2010
Callout & Web Surveys - Part 2 - Normal Fans and Extreme Fans
Exploring the Differences between Callout and Web-based Music Surveys
Part Two – Normal Fans and Extreme Fans
For insight as to why music research results can be so different between Callout and station Website based hook tests, Kelly Music Research is conducting a series of nationwide tests focusing on different components of each. We recently completed a telephone perceptual survey of over 300 random music radio station P1 listeners aged 18 to 54 in the top 25 markets including land line and cell phone households. These are typical Callout research respondents who participate in music surveys over the telephone. We asked these listeners a series of questions about radio listening and their communication with radio stations.
At least part of the reason music research results differ between radio stations’ Website based surveys and Callout surveys can be traced to the types of listeners who make up each panel sample. Callout panels have more “Normal Fans,” while Web based panels have more “Extreme Fans.” And in most radio audiences, our research shows that Normal Fans outnumber Extreme Fans by about 25 to 1.
In Kelly Music Research’s national telephone survey, only 4% of music station P1’s exhibit Extreme Fan behavior. 96% of those polled can be described as Normal Fans.
Normal Fans are P1’s, often referred to as “Passive P1’s”. Your radio station is their first choice. Yours is their “go-to” station for their favorite music. Normal Fans can be very heavy users of your station, but they will tune away if you play, say or do something that they don’t like. However, Normal Fans are not complainers. They use Radio like they use electricity or any other utility. They turn it on, they change stations and they turn it off without giving it any more thought. Normal Fans account for the lion’s share of every station’s audience and ratings. Normal Fans are the silent majority.
Extreme Fans are different. Also known as “Actives,” Extreme Fans have a strong desire to become more involved with radio stations. Reasons for that desire vary. For example Extreme Fans might have a higher than average passion for music, often called “Purists”. Or they may have an infatuation with air personalities or core artists. Some Extreme Fans have a fascination with the show biz nature of Radio and want to be a part of it. Extreme Fan behavior would include heightened interest in radio station contests, texting jocks and making requests, following personalities on Twitter and Facebook, showing up at promotions and voicing their opinion about programming. Whatever their motivation, Radio’s Extreme Fans feel a much higher emotional connection, and even worship of radio stations, than Normal Fans do. Extreme Fans are the vocal minority
Other industries have Normal and Extreme Fans too. Take the Sports and Movies industries as examples. Professional sports franchises have Normal Fans who watch games on TV and occasionally attend home games. However, Extreme Fans go to all the home games as well as some away games. Extreme Fans comment in player and coach blogs, argue game strategy on call-in shows, know all the players first and last names, where each went to college, and whether players are married and have kids. Extreme Fans paint their faces with team colors, make signs and show up at the airport wearing team uniforms to see their players off and welcome them home from road games. Similarly, in the Movie industry, Normal Fans will go to the theater to see the Star Trek movie. Extreme Fans dress up like the characters and attend Star Trek conventions.
Extreme Fans are often characterized as “groupies” or “obsessive” by Normal Fans. Like Sports and Movies, Radio should not ignore Extreme Fans; however catering to Extreme Fans can alienate the much larger base of Normal Fans. And it is dangerous to think that Extreme Fan opinion and behavior is representative of Normal Fans.
In a simple example, if a DJ says that he or she likes a particular song or artist, that opinion can influence the way a listener scores a song being tested. Normal and Extreme fans often respond differently in music testing. Careful not to be negative about anything the station does, Extremes tend to score everything in degrees of positive and can be biased if they are also a fan of the artist. In Callout surveys, if a Normal Fan doesn’t like a song, he or she tends to say so honestly. In comparative surveys, Callout respondents are not as easily influenced by outside factors such as loyalty to air personalities, stations or artists. There is reduced likelihood for Extreme Fan bias in Callout because respondents are not informed which radio station is fielding the survey and artist names are not revealed.
It is important to note that not all radio station Website visitors are Extreme Fans. Normal Fans visit station Websites too. The #1 reason Normal Fans visit music radio station websites is to listen live to an internet stream. Normal Fans may also glance at pictures, check out weather, traffic or a podcast. But for the most part, Normal Fans don’t participate in radio contests and don’t go out of their way to join advisory panels or voice their opinions.
Over the years, technology has changed our communication apparatus but behavior patterns of Normal and Extreme Fans has not. Prior to cell phones and computers, Extreme Fans jammed request lines, attended remotes and wrote letters to DJ’s or PD’s. Meanwhile, the bulk of the audience, the Normal Fans, listened quietly. Today, Normal Fans still listen quietly. And Extreme Fans text, upload, e-mail, follow on Twitter and join station databases. The key to successful research is achieving the proper balance of Normal and Extreme Fan opinion.
Next time, the “Prize Pig” factor.
Part Two – Normal Fans and Extreme Fans
For insight as to why music research results can be so different between Callout and station Website based hook tests, Kelly Music Research is conducting a series of nationwide tests focusing on different components of each. We recently completed a telephone perceptual survey of over 300 random music radio station P1 listeners aged 18 to 54 in the top 25 markets including land line and cell phone households. These are typical Callout research respondents who participate in music surveys over the telephone. We asked these listeners a series of questions about radio listening and their communication with radio stations.
At least part of the reason music research results differ between radio stations’ Website based surveys and Callout surveys can be traced to the types of listeners who make up each panel sample. Callout panels have more “Normal Fans,” while Web based panels have more “Extreme Fans.” And in most radio audiences, our research shows that Normal Fans outnumber Extreme Fans by about 25 to 1.
In Kelly Music Research’s national telephone survey, only 4% of music station P1’s exhibit Extreme Fan behavior. 96% of those polled can be described as Normal Fans.
Normal Fans are P1’s, often referred to as “Passive P1’s”. Your radio station is their first choice. Yours is their “go-to” station for their favorite music. Normal Fans can be very heavy users of your station, but they will tune away if you play, say or do something that they don’t like. However, Normal Fans are not complainers. They use Radio like they use electricity or any other utility. They turn it on, they change stations and they turn it off without giving it any more thought. Normal Fans account for the lion’s share of every station’s audience and ratings. Normal Fans are the silent majority.
Extreme Fans are different. Also known as “Actives,” Extreme Fans have a strong desire to become more involved with radio stations. Reasons for that desire vary. For example Extreme Fans might have a higher than average passion for music, often called “Purists”. Or they may have an infatuation with air personalities or core artists. Some Extreme Fans have a fascination with the show biz nature of Radio and want to be a part of it. Extreme Fan behavior would include heightened interest in radio station contests, texting jocks and making requests, following personalities on Twitter and Facebook, showing up at promotions and voicing their opinion about programming. Whatever their motivation, Radio’s Extreme Fans feel a much higher emotional connection, and even worship of radio stations, than Normal Fans do. Extreme Fans are the vocal minority
Other industries have Normal and Extreme Fans too. Take the Sports and Movies industries as examples. Professional sports franchises have Normal Fans who watch games on TV and occasionally attend home games. However, Extreme Fans go to all the home games as well as some away games. Extreme Fans comment in player and coach blogs, argue game strategy on call-in shows, know all the players first and last names, where each went to college, and whether players are married and have kids. Extreme Fans paint their faces with team colors, make signs and show up at the airport wearing team uniforms to see their players off and welcome them home from road games. Similarly, in the Movie industry, Normal Fans will go to the theater to see the Star Trek movie. Extreme Fans dress up like the characters and attend Star Trek conventions.
Extreme Fans are often characterized as “groupies” or “obsessive” by Normal Fans. Like Sports and Movies, Radio should not ignore Extreme Fans; however catering to Extreme Fans can alienate the much larger base of Normal Fans. And it is dangerous to think that Extreme Fan opinion and behavior is representative of Normal Fans.
In a simple example, if a DJ says that he or she likes a particular song or artist, that opinion can influence the way a listener scores a song being tested. Normal and Extreme fans often respond differently in music testing. Careful not to be negative about anything the station does, Extremes tend to score everything in degrees of positive and can be biased if they are also a fan of the artist. In Callout surveys, if a Normal Fan doesn’t like a song, he or she tends to say so honestly. In comparative surveys, Callout respondents are not as easily influenced by outside factors such as loyalty to air personalities, stations or artists. There is reduced likelihood for Extreme Fan bias in Callout because respondents are not informed which radio station is fielding the survey and artist names are not revealed.
It is important to note that not all radio station Website visitors are Extreme Fans. Normal Fans visit station Websites too. The #1 reason Normal Fans visit music radio station websites is to listen live to an internet stream. Normal Fans may also glance at pictures, check out weather, traffic or a podcast. But for the most part, Normal Fans don’t participate in radio contests and don’t go out of their way to join advisory panels or voice their opinions.
Over the years, technology has changed our communication apparatus but behavior patterns of Normal and Extreme Fans has not. Prior to cell phones and computers, Extreme Fans jammed request lines, attended remotes and wrote letters to DJ’s or PD’s. Meanwhile, the bulk of the audience, the Normal Fans, listened quietly. Today, Normal Fans still listen quietly. And Extreme Fans text, upload, e-mail, follow on Twitter and join station databases. The key to successful research is achieving the proper balance of Normal and Extreme Fan opinion.
Next time, the “Prize Pig” factor.
Subscribe to:
Posts (Atom)
