Monday, December 14, 2009

Callout and Web Based Research - Exploring the Differences

An Open Discussion
Differences between Callout and Web-based Music Surveys
Why are the results so different?
Part One – We need answers!

The programming minds that steer the content of the radio station are in a music meeting. It’s decision time. Your Callout research says a song on the playlist is negative and driving listeners away. But your Web survey says listeners like the song. Which one is right? Regardless of how you deal with these dilemmas, you might also ask yourself, “Why?” Why does Callout data look different from Web research data? There are many opinions around Radio, but little has been studied or written on the subject.

We think this topic is worth exploring and talking about. Kelly Music Research is taking a closer look at Callout and Web surveys. We also want to create an open forum among our broadcasting brethren and encourage your participation. In Radio we have a wealth of brainpower among programmers, consultants and researchers. Collectively, we can certainly create an interesting conversation. And who knows… maybe discovery. Over the past several months we’ve begun laying the groundwork for this conversation. We’ve done parallel music tests using different methods and approaches. Many of you have actual experiences we hope you’ll share. This is an open conversation that we hope will be informative and productive. Incidentally – full disclosure - Kelly Music Research does both traditional telephone callout and we have a computer survey tool. We are also funding this project ourselves. There is no third party interest. We don’t have an agenda other than to facilitate greater understanding and perhaps discovery for the benefit of Radio.

Responding to the ever changing desires of listeners is tricky but essential in a competitive marketplace. Traditionally, Callout has been a potent and critical weapon in the arsenals of America’s most successful radio stations that play current music. Callout is not cheap. Many programmers want Callout to be the very last tool to be stripped away in budget cuts. In competitive situations, PD’s and consultants feel that Callout can tip the scales. Success or failure. A bonus or a dismissal notice.

Like it or not, current economic conditions have forced Radio to rethink everything. Economic recovery will be gradual, so some stations are forced to consider less expensive tools. And Web research is cheaper. Web based research has not yet achieved the same level of credibility among programmers that Callout enjoys. Most programmers with the luxury of both still have more confidence in their Callout data. Some of those being forced to drop traditional Callout in favor of less expensive Web based research are lamenting their predicament and say things like, “Web data is not real research.” On the other hand, proponents of Web based research enjoy the ability to use a computer as the hook delivery and scoring apparatus. A computer will play the hooks for the respondent in higher audio fidelity than over the phone. The computer also allows for visuals such as pictures and text display which can include artist name, song title, CD cover, etc. And on the computer, the survey can be done at a time convenient to the listener. Conversely, most telephone Callout research is conducted weekdays between 6pm and 9pm without mention of artists or titles. Clearly, both methods have their limitations. And the use of certain elements could introduce bias that can influence the respondents and affect song scoring.

Particularly now, broadcasters are hopeful for solutions that provide both reliability and affordability. Unreliable data is ultimately more costly if it makes the station more vulnerable and sacrifices audience, ratings and revenue. In this discussion we will research the research. In the coming weeks, our field studies will focus on sample recruitment, scoring apparatus and potential biases that might affect test results. As of this writing, our field work is continuing in markets across the country. We’re not finished yet, however what we’ve seen so far is very interesting. Whether you are new to the business or a grizzled veteran, if you have an interest in programming or research, we believe you will find this discussion informative and thought provoking. Chime in with your own thoughts, experiences and points of view. Your participation in this conversation is strongly encouraged. Radio will benefit.


  1. I use RadioTraks---I e-mail MY P-1' do I know they are P-1? They belong to my e-mail club and they respond to offers and they LISTEN to the station. I do a survey every 2 weeks and my in-tab sample is NEVER less than 200--and gets as high as 400-500. This means I can have sample sizes in subcells 18-24 Women, for example--that are larger than some stations entire callout sample.

    Callout is farmed out-there is zero quality control and the station is totally at the mercy of the research company. Burn may show faster with web, but we can allow for that. Ability to do a quick perceptual is also great.....OH....did I mention that it is FREE?
    ZERO cash, no barter.

  2. Call Out is just once piece of the puzzle. Albany, NY recently became a cell phone only Arbitron market. If call-out could include cell phone only listeners, I think that would be a key to making it that much more accurate.

  3. Very interesting, but please note, in your case, nothing is free. Record companies are paying for your research.

    Also, in the interest of being open, please identify yourself to alleviate comment edits or non-published comments.

  4. It is true that record companies are paying for I repeat--IT IS FREE to me. Supposedly, they also get to look at the research, which is fine with me--as long as they are not influencing the results (which they cannot). It actually gives a PD ammo when labels are trying to promote a stiff...maybe nothing is "free", but "free to me" is all I care about, frankly. I would gladly pay for this data if I had the budget. The truth is that if they ARE looking at the data, it is not often--as they are constantly asking ME how their own records are performing in callout.

  5. Using our web database is what we expect to be the most dependable, but with the way our database is gathered.. we dont not get the amount of respondents that we need consistantly for the demo that drives our music. I do also use Kelly research, but I can not get specific breakdowns with what I get and it's left as a big pool of 16-34 females (opposed to a breakdown to day part young from older). Also I'm trying to understand why our top songs differenciate with Kelly to internet at times.

  6. Callout gives a much wider view of a song, web research gives a narrow view of your core audience's tastes. Both have their upside, both have their downside, however, I wouldn't dream of doing callout in 2009.

  7. Jeff - You raise a common concern about why your top song scores in Callout and Web are so different. And more narrow age demo scores are not statistically reliable for your current Callout program.

  8. A few Rock PD's and consultants checked in and cited examples from web based surveys where established brand name bands like AC/DC and System of a Down tested 100% familiar and very positive within a week or two, while Callout revealed much lower scores. Country stations noted that while the new Miley Cyrus scored well in Callout, web survey scores were negative. In each of these cases, the web survey displayed the artist name as the hook played. Are they scoring the artist or the song? The artist's image and "brand" may influence scoring.

  9. I had the opportunity to monitor a night of call-out. It was fascinating and frightening at the same time. Several times, respondents just gave up half-way through. Hearing only a few hours, I wondered how we were going to get 40 respondents.

    I also couldn't help but notice a general sound of apathy in many of the caller's voices - like they wondered why they even picked up the phone.

    My experience monitoring was before internet and cell phones were prominent and caller ID was still expensive. I can't imagine how long it would take to get 40 respondents today. How do they get through?

    Since then, I've wondered if that panel is indicative of anything other than people who are abnormal enough to answer a telemarketing call?

    I've also wondered about internet panels. But 1000 respondents is easier to smooth over in my head than a rolling 40.

    I agree that internet panels are too hip for the room and we programmers need to use the data with that knowledge. I use the data as thought starting - or trend watching.

    That night of monitoring was my Clarice Sterling moment ... as I heard my own screaming lambs for years for years to come.