Nothing to Smile Sheet About

I’m admittedly bad at titles, so I’ll summarize what you’re about to read:

Those smile sheets most of us use to “evaluate” an educational event are nothing to smile about.

Refresher: “Smile sheets” are those questionnaires handed out (or worse, left on tables or chairs) for attendees to fill out at the end of an educational session. They usually ask for a numerical rating of the session and/or presenter and are generally limited to less than a page. They’re supposed to help us determine whether the session and its presenter(s) delivered on the promise the attendees felt was given.

Smile sheets don’t work.

• They’re overused
• They’re poorly written
• They’re inaccurate

Overused: We’re busy people, no doubt about that. To be more efficient, we take every possible shortcut, including making a generic smile sheet and using it for all of our events. What’s worse, when smile sheets are the only means of soliciting feedback and results about our programs, we are asking them to do something they’re not designed to do: assess final learning and application of that learning to the workplace.

Poorly Written: Because they’re so generic, they don’t have the specificity needed to generate helpful responses. Asking an attendee if the presenter was knowledgable about the topic or if they will be able to apply what they learned back on the job is useless. Forming useless questions in perfect ways won’t help, either.

Inaccurate: Generic, poorly written smile sheets won’t provide accurate feedback. And if you ask the smile sheet to do more than it can, you’ll believe you’re getting guidance from your members about future programs that they’re really not providing. Smile sheets are not accurate for determining whether learning occurred. It won’t even tell you whether the session had value for those attending.

Why not? Because that’s not what a smile sheet should be designed to do.

What should a smile sheet do? A smile sheet can be very useful for gathering feedback on:

• Location: Was the hotel/conference center conducive to learning? Was the city a desirable place to be for this topic, theme, or session? Why or why not? Answers to location questions can help you determine if you leveraged the local environment sufficiently for your event. If people are going to travel to attend your conference or other events, make it worthwhile. Then find out if it was.
• Content: Asking the attendees to list three things they learned that they will be able to apply back in the “real world” (whether the “real world” is at work, at home, or in other actvities) will help you determine whether any key information was new to the general group, whether any of the content “stuck,” and if what was presented was done in a way that made people believe they could do something with what session covered.
• Format: Were they engaged the entire time? If not, why did they disconnect? Were any questions they had answered?
• F&B: If your sessions include providing refreshments of any sort, ask about them. Was there enough variety? Were special dietary needs met?
• General: If you’ll be using the same location in the future, you should ask about the room and service: was it comfortable? Were they able to stay focused? Were there distractions (for example, did hotel staff disrupt the session when they came in to clear the refreshments?)?

The bottom line is that people will have an immediate reaction to their environment (“That was great! The food was awesome!”), a general idea about what they’ll find useful (“I loved that idea about using Twitter to send my cookbook customers tweets about what I’m ordering at the restaurant for dinner tonight.”), and whether they found the session interesting or not (“That guy seemed to know his stuff, but sitting for 90 minutes and hearing him talk just couldn’t hold my attention, no matter what an expert he is.”).

To find out what your members end up applying back in the real world, conduct separate, follow-up surveys or interviews.

Basing your decisions about content on smile sheets will lead you down the wrong path. And basing your ROI (return on investment) will take you down even worse roads.

What Was That Noise?!?!

Remember that alarm that went off last year? The loud one that seemed to wake up PD and education leaders in associations everywhere? Remember scrambling to figure out alternatives to face-to-face meetings and educational events because you worried that members might not be able to attend them?

Even those most reluctant to move into online learning saw real value in developing elearning for their members. Many actually implemented new initiatives or ramped up existing alearning programs.

But what happened when the alarm bell went silent? Did your alearning strategy go back on the shelf?

If you have a hard time justifying the value of elearning outcomes to your association leaders or members, here’s another piece of evidence for you:

Relative to learning efficiency, according to Brandon Hall, online learners are twice as efficient in learning times as ILT learners. In other words, depending on the structure of the course, online learners may be able to acquire the needed information in half the time they would in a traditional classroom. If a learner can jump ahead or skip around in an online module to get exactly what they need, instead of sitting in a classroom for a designated amount of time, he or she will potentially save the organization time and money.    — From the e-Learning Guild’s report, “Building the Case for eLearning”

[ILT = "instructor-led training" or classroom instruction]

Because of all the focus on collaborative and social learning, we sometimes overlook the fact that our members sometimes have very good reasons to learn individually rather than in a group, in person, face-to-face, sitting at the feet of an expert instructor.

Yes, it’s true!

If your association membership includes competing entities (for example, medical supply companies that compete with each other for the same customers), then you might have attendees who would prefer not to reveal what they don’t know. In this case, they might avoid attending a face-to-face session that they *would* take online, privately.

If your association membership includes individuals within the same organizations but at different levels of responsibility, online learning can appeal to those who’d prefer to review basic information privately, online, than to expose their need publicly at a face-to-face event. For example, if I’m relatively new to my industry and I know that I’d benefit from attending a certain session, I might be reluctant to register because I don’t expose my knowledge gap to my direct reports or peers. Even if I attend, I might be less willing to fully participate, which won’t result in the best learning outcome.

If your association members possess a range of educational and experiential levels, providing face-to-face events that adequately cover all needs is impossible. Providing online learning that allows everyone to register and to skip around the content as needed, then you will have leveraged the online event to meet the maximum number of members for the best financial investment, resulting in targeted learning for the attendees. In this case, it’s important that you avoid emphasizing “completion” of the course or module; instead, suggest skipping content that’s already familiar and focusing on what’s needed. Even better, design the content so they only need to access and pay for what’s needed — i.e., create brief segments based on specific topics or a problem that needs solved, versus developing a broader module or course.

You can do this.

You need to do this.

If you don’t do this, your members will eventually drop out, fading away just like the blaring of that persistent alarm a year ago.

Valuable LMS Advice

Just discovered the Upside Learning blog and knew right away I wanted to share what Amit has to say about selecting an LMS.

Take a look at Five Things Not to Do While Selecting an LMS.  Great advice from someone who knows the tech side of elearning much better than I do!

Education as a Commodity?

When I first saw “edutizing” mentioned as a way of building business (see Judith Lindenau’s post: http://www.realtown.com/Judith2/blog/edutizing) I had a gut-level, negative feeling about it.

Now I’m thinking… maybe that’s what most trade associations are doing anyway — using education to help members buy into membership.

What do you think? Are we “edutizing”?

Thank you, Bill!

Bill Brandon, editor of e-Learning Guild’s Learning Solutions e-magazine, published a review of aLearning: A Trail Guide to Association eLearning that called the book “a stunning success.”

Thank you, Bill! As someone posted to my Facebook wall after reading the review, “If Bill says it, it must be so!”

If you’re not familiar with the e-Learning Guild, consider joining — their basic membership level is designed for the types of budget restrictions common to non-profits and associations — it’s free. Their next level, which provides a terrific value for the money, is just $99/year. Their focus is on elearning development, with a mission to provide specific, practical, applicable, implement-able (!) information and training.

Even if you’re not a techie, it’s important to know enough about the tech side to be able to ask a question that the techies will understand (and translate into their own geek-speak). The eLearning Guild can help you with that.

Most of their members are Web developers, instructional designers, project managers, graphic artists, and others engaged in building elearning, either in-house for large organizations (including large associations) or within vendor companies.

Over the years, they have expanded their conference tracks to include management and strategy, so those of us who oversee online learning can benefit from the practical workshops and tutorials they provide.

If you’re considering hiring a company (for Webinars or custom courseware) to develop elearning for you, or if you’re looking for a good LMS or LCMS, the e-Learning Guild’s Annual Gathering (coming up again in March of 2010) is a MUST. You’ll learn more and make more contacts for potential vendors here than at any association-specific event.

I’m not just saying this because Bill had such great things to say about my book. And he does say this:

This is one of the best-executed books I have seen in some time on the topic of learning strategy. In my opinion, every association should have a copy of this to refer to during creation or revision of the professional development curriculum.

It will be particularly useful to associations with small staffs (and no professional development staff). It will also be useful to vendors who have little or no experience or insight into the challenges faced by associations as they adopt e-Learning. Finally, consultants on the staff of associations, who are more and more often asked to help their member organizations create a comprehensive learning strategy that includes online and social media components, should definitely have a copy of this.

If Bill says it, it must be so!

Thank you, Bill! And thank you to those of you who have already purchased your copy – I’m happy to help and you know where to find me if you have questions not answered in the book!