Thursday, July 28, 2011

Chicago School Board Flunks Research 101

Screener Imparts Lesson: What Not to Do

By Marc Dresner, IIR

This week’s rant may or may not be brought to you courtesy of the Chicago Board of Education. We don’t know...

Few things irritate me more than being subjected to a poorly designed survey. Sadly, I didn’t qualify, or this post would probably be much longer.

Instead, the screener will have to do, and I’m sure you’ll agree it speaks volumes about the instrument that likely followed, not to mention the irresponsible knuckleheads who created it.

Before I rip into the Chicago Board of Education as promised in the title of this post, I’m obliged to note that I’ve been unable to determine that CPS is beyond a doubt the party responsible for the survey in question, although I suspect as much.

When I actually managed to get a live human being on the phone, my queries at the city, county and even the state level were predictably met with either confusion and/or belligerence, and I was invariably sent on a snipe hunt. E.g., “What you need to do is call so-and-so in our public affairs department at this dead-end number. They’ll do a much better job of frustrating you.”

But more often than not I just left messages—lots of messages—that went unanswered.

What I can tell you is that someone purporting to represent CPS called me two weeks ago and asked me to participate in a survey, to which I agreed and was subsequently barred after TWICE submitting to a ham-handed screener.

Why, oh why, didn’t I just lie? At least that way I might’ve gotten past the screener and to the bottom of this mystery.

I knew something fishy was afoot when my caller ID could not identify the source, except to tell me that this was an Illinois call with an 800 prefix.

Curiosity got the better of me. Upon answering the phone, I was greeted with an IVR screener and a vague invitation along the lines of: “We are conducting a brief survey for Chicago Public Schools. Would you please take a moment to answer some questions? If ‘yes,’ please say ‘yes’ or press one. If ‘no,’ please say ‘no’ or press two.” (In Spanish: To proceed in Spanish, press three.)

My sense of civic responsibility prompted me to say “yes,” at which point the fun began.

“I’m sorry, but I didn’t understand you. Please say ‘yes’ or press one to proceed.” (At this stage, they dropped the “no” option, presumably because “no” is typically expressed by the respondent hanging up the phone.)

So I pressed one. “Thank you for participating. We will now ask you a series of questions. Are you a Cook County resident? Please say ‘yes’ or press one to proceed.”

I said yes. “I’m sorry, but I didn’t understand you. Please say ‘yes’ or press one to proceed.”

I pressed one. “I’m sorry, but I didn’t understand you. Please say ‘yes’ or press one to proceed.”

I pressed one, again. “I’m sorry, but I cannot complete your request. Please hang up and try back later.”

(Of everything, that last message led me to believe that the survey was the genuine article because I’ve heard it time and again in my dealings with our local government.)

So—apologies for the acronym, as this is a family-friendly research industry blog—but WTF?!? My request? Try back later?!?You called me.

Not two minutes later, the phone rang again!

Now they had my complete attention, and you bet I picked up. This time, I made it past the Cook County resident question to the following: “Do you reside in the City of Chicago? Please say ‘yes’ or press one to proceed.” I pressed one.

“Are you the parent of a child who currently attends public school in Chicago? Please say ‘yes’ or press one to proceed.”

I said no—even though it wasn’t an option on the menu—and was then asked if I was the grandparent of a child who currently attends public school in Chicago. Nope.

“We’re sorry, but you do not qualify for this survey. Thank you.” Click.

How rude! The only thing worse would’ve been for them to substitute a four-letter expletive that rhymes with “thank” before hanging up on me.

As a childless blogger and taxpayer, I was shocked by this unceremonious rejection. So I dialed the number on my caller ID to vent my outrage. But as you might’ve already guessed, the number I dialed did not accept inbound calls.

I spent the next 15 minutes fuming, stroking my phone like a loaded gun and praying for them to call back. Alas, it wasn’t meant to be.

Technical difficulties aside, this was a frightening screener for several reasons, not the least of which was the sheer stupidity of its design.

Why bother asking me if I live in Cook County when what you really want to know is if I live in Chicago? You can’t live in Chicago without living in Cook County, so the first question is moot.

Moreover, why not just tell me upfront that you are conducting a survey among parents and grandparents of children who currently attend public school in Chicago? It would’ve spared me the trouble of taking the screener in the first place.

And who exactly is the “we” in “We are conducting a survey for Chicago Public Schools”? The sponsor might be implied, but never actually identified itself.

Nor did the sponsor provide a phone number or URL so that a respondent might verify that this is a legitimate source before answering questions about a relatively sensitive topic: a person’s child or grandchild.

And then there’s the matter of the mysterious caller ID. If you’re calling on behalf of CPS, this should be clear on the caller ID so that I know immediately who the source is, and so that I don’t ignore the call because I don’t recognize the source, which may not only have a deleterious effect on the response rate, but also probably introduces a response bias: What sort of person in this day and age answers a phone call at home from an unfamiliar and borderline suspicious source? (Yes, I am aware that I now fall into this category.)

Maybe I’m too sensitive, but as someone long affiliated with the research industry who cares about its viability, respondent abuse agitates me. They don’t grow on trees.

The technical glitches, the bumbling, amateurish design, the inadvertent or perhaps even deliberate concealment of the sponsor, and the abrupt kiss-off at the end not only aggravated me; it all worried me.

The telephone is still a staple modality for academic and government survey research. This screener in my opinion is a weapon of mass destruction. It erodes people’s trust in survey research, which for telephone is already abysmal thanks to telemarketers, suggers and push pollsters.

But what really chafes me is that I’m confident that this folly was in all likelihood sponsored by my local government—which means my tax dollars paid for it—on behalf of our public schools.

Anyone who has lived here and has dealt with the city would probably agree that this screener screams Chicago. Cumbersome, confounding and with a callous disregard for people’s time and feelings, it’s got all of the hallmarks that endear us Chicagoans to our bureaucrats and politicians.

It’s just so irresponsible. And can you image how flawed and potentially misleading the results will be? I shudder to think what they’ll do with them.

The next time you hear some high-fallutin' ivory tower type bash commercial research, please let me know. I’ll gladly pay for a one-way bus ticket to the windy city, where the weather isn’t the only thing that blows.

ABOUT THE AUTHOR
Marc Dresner is an IIR USA communication lead with a background in trade journalism and marketing. He is the former executive editor of Research Business Report, a confidential newsletter for the marketing research and consumer insights industry. He may be reached at mdresner@iirusa.com. Follow him @mdrezz.

No comments: