aLearning Blog

Online Learning for Trade Associations

Archive for the ‘Measuring Results’ Category

How Time Flies…

Posted by Ellen on January 27, 2012

…when you’re having fun, right?!?

And we have been having fun here at the aLearning Blog! Suddenly, it seems, we’re publishing our 250th post and celebrating five years.

Yep, five years. And so much has changed!

When aLearning published its first post back on January 27, 2007:

  • no LMS systems (that we know of at the time) were designed especially to meet the needs of associations and nonprofits
  • few (if any) research endeavors about online learning focused on associations and nonprofits
  • few (if any) organizations bothered to survey association learning leaders to find out what we’re doing in the field and how things were going
  • the number of association-specific blogs could be counted on the fingers of one person’s hands
  • social learning and virtual learning environments were mysterious, hocus-pocus, scary entities

A lot has changed over just five changes of the seasons, hasn’t it?!?

Top 100 aLearning Blog Posts

To celebrate this milestone, we’ve compiled an ebook of our Top 100 aLearning Blog Posts. Just skimming through these selections made us realize how quickly the elearning sands shift, affecting the landscape, even moving the horizon.

At over 200 pages, this compilation brings together in one place the best — and most controversial — writing from the aLearning Blog. We’ve included most comments (the fine print is that we’ve deleted pingpacks, backtracks, and outright sales pitches) and are proud of the attention the aLearning Blog has garnered over the years by elearning and education experts.

To Get Your Copy

We’ve made this e-publication very affordable at just $5. To order, go to and click the “Buy Now” PayPal button. You should be able to read this PDF from any device with a PDF reader (such as Adobe Reader).

Special Offer

If you’ve purchased aLearning: A Trail Guide to Association eLearning, we’ll send you a copy of the Top 100 Posts for free. Just send Ellen an e-mail at and attach an electronic copy of your Lulu receipt, and we’ll send you the Top 100 Posts by return e-mail. We appreciate your support and are happy to say “thank you” in this small way.

Thank You!

Posted in aLearning Strategies, aLearning Surveys, aLearning Trends, Conferences, eLearning Marketing, eLearning Resources, Financing eLearning, Justifying aLearning, Learning in General, LMS, Measuring Results, Online Learning in General, Social Learning, Webinars | Tagged: , , , , , , , , , , , , , , , , | Leave a Comment »

The Danger of Data (Even When It’s Accurate)

Posted by Ellen on January 6, 2012

You know I love data. You know I believe in the power it has to convince TPTB (The Powers That Be) in the need for elearning, for improved instructional design in face-to-face training, and all other related matters.

We worry a lot about getting our data wrong and about the dissemination of incorrect information. But when’s the last time you worried that accurate data could lead you — or your association leaders or your members or your other constituents — in the wrong direction?

Yes, it’s possible.

Here’s a perfect example, courtesy of Mark Twain:

“In the space of 176 years the Lower Mississippi has shortened itself 242 miles. Therefore… in the Old Silurian Period, the Mississippi River was upward of one million three hundred thousand miles long and … 742 years from now the Lower Mississippi will be only a mile and three-quarters long…. There is something fascinating about science. One gets such wholesome returns of conjecture out of such a trifling investment of fact.”

Moral of Mr. Twain’s story? Use facts and data wisely.

If you’re projecting the results of something, project something that’s realistic. If you’re drawing a causal relationship to examine what might have caused X, Y or Z to have happened, then do so logically.

Accurate data, wielded irresponsibly, could be more disastrous than not having data at all. Remember that.

Posted in aLearning Strategies, Justifying aLearning, Measuring Results | Tagged: , , | Leave a Comment »

What Would Your Members Say?

Posted by Ellen on August 30, 2011

When you spend a lot of time in an RV on the road, visiting different places, staying in various campgrounds, parks, and RV resorts, you realize what your expectations can be, based on the name of a place.

A campground, for example, traditionally includes places for tents, which means a bathroom facility that also includes showers (unless the campground is considered “rustic,” in which case, you can expect no such facilities to be on the grounds). At the other end of the scale, a resort is usually a meticulously landscaped property that often includes a pool, spa, and other amenities — horseback riding, or a clubhouse with activities and game room. We’ve stayed at resorts that only allowed certain types of RVs (yes, it’s permitted by law), and cost more than a room at a bed and breakfast.

This is important background for the point I’m going to make. Stay with me here.

We’ve stayed at a couple of places that were advertised as “resorts” and ended up being — on the high end — family campgrounds with a dilapidated miniature golf course and a campfire ring as the “resort amenities.” At another “resort,” the pool turned out to be above-ground, the horesback riding option cost more than a night’s stay, the “lodge” was being used as a private residence…

We don’t mind rustic as long as the facilities are usable and the power is reliable (faulty power feeds can destroy the electrical circuitry in an RV… an expensive repair), so we considered the experience an adventure and spent our spare time watching frustrated RVers pull in only to leave again. Yesterday we watched a man who’d come with two kids photograph the broken swingset, unsafe picnic tables, and “Out of Order” sign on the ladies bathhouse door. No doubt he’d thought the kids would be entertained with the hiking (we’ve yet to find the trail), volleyball (or a net), cookouts (don’t see a grill anywhere)… Today they’re gone.

So the first message in all of this is:

  • Advertise accurately. Don’t advertise something you’ve stopped doing or plan to do. Promote only the things your members can expect from you on a regular basis.
  • Think about your name. What expectations are you setting for potential members? If call yourself a “resort,” people will expect certain things from you and they’ll leave, dissatisfied, if you don’t. What do your new members expect from you? Are you delivering on your promises?
  • Don’t make excuses. When somebody says, “You call this a resort?!?” don’t say, “Well… eventually we’ll have cookouts… and hiking… and a spa… and an in-ground pool… it takes awhile to get there.” Nobody wants to hear their timing is bad and they’re missing the good stuff. Don’t say, “We just took over and the place was such a wreck it’s going to take awhile to get it into shape.” Nobody wants your whining. They just want what you promised them and they’re understandably upset when you can’t deliver it.
  • Accentuate the positive, as an old lyric goes. If the place is rustic and out of the way, say so. People who like “rustic” will stay and say nice things about the place, instead of leaving frustrated, angry, and feeling they’ve been had.

Which leads me to the next part of this analogy. Out of curiosity, my husband looked up this particular “resort” on the Web to see what others had to say about it, and post our experiences. We weren’t surprised to read a list of complaints by those who’d come expecting one type of experience but left upset. We posted a cautionary but accurate review: yes, the property has its issues, but we like that it’s quiet and off the beaten path; those looking for a “resort” experience ought to look someplace else.

The RV world has many forums and bloggers. RVers are an honest lot, and they like to share their stories. When RVers meet, they swap suggestions about attractions, restaurants, and places to stay. Word of mouth — especially via the Internet — is loud and long.

So — here’s the real question: if there were an online forum for reviewing associations, societies, institutes, councils, and other non-profits, what would former and current members be posting about you?!?

  • Does your membership brochure match what they’ll actually get? Or are you setting expectations you can’t meet?
  • Are your dues and fees fair? Or are you leaving members wishing they’d saved their money for something else?
  • If they could post a comment on a forum about the associations they’ve been members of, what would they say about yours? Would they recommend others join? Or would they warn other people against joining? Why?

All of this could be true about any aspect of your organization: what would they say about volunteering? Membership benefits? Educational experiences?

  • “I spent three hours a day of my own time on Project X for this association and then they completely ignored our group’s findings and recommendations… that’s the last time I do that!!”
  • “For $200 a year I get a magazine. Everything else is geared to attending face-to-face sessions I can’t afford to attend. This is my last year as a member. I can get a magazine for a lot less money.”
  • “They have some of the best educational sessions I’ve ever attended. If you haven’t been to one, go! They’re a little pricey, but you’ll get a lot out of them.”

There might not be an “associations forum” where people post their reviews of their membership experiences the way they do RV campgrounds, parks, and resorts, but they are sharing their experiences.

What are they saying? And what can you do about it?

Posted in aLearning Strategies, Conferences, eLearning Marketing, Learning in General, Measuring Results | Tagged: , , | Leave a Comment »

Quick Clicks

Posted by Ellen on July 11, 2011

Once again, aLearning is pleased to be able to save you enormous time and energy by perusing hundreds of blogs and resources to summarize those likely to be of particular interest to you. If you have any links that have been especially insightful or helpful to you, please feel free to add them to the comments section or send them to me directly for inclusion in a future Quick Clicks post.

These are in no particular order…. so scroll the entire post to be sure you won’t miss anything.


LMS? LCMS? What’s the Diff?

Thanks to this article by Nic Hinder, courtesy of the “Funderstanding” ( site and brought to our attention by Julien R. Barlan, Research Associate at Funderstanding, we’re all a little bit clearer on what distinguishes the two types of systems, the features that most often overlap, and when each system is put to best use (and by whom). That’s a lot to cover, and the article does it clearly and succinctly. You can find it here.


iSpring Free Update

If you’ve browsed our aLearning Fundamentals tutorials lately, you’ve noticed we went from a pure PowerPoint download to using iSpring to compile the PowerPoint for easier access and viewing. Though we use iSpring Presenter for most of the tutorials, we want to pass along this update regarding the iSpring Free version, which is available to non-profits for — you guessed it — free (although the Presenter version is not that expensive, compared to its competition).

According to an e-mail from Tanya Mosunova at iSpring:

iSpring Free 5.7 can generate Flash content that is compliant to SCORM 2004 R3, the most popular standard used in today’s eLearning. This means that with iSpring, courses created in PowerPoint can become compatible with Learning Management Systems (LMSs) that also support SCORM. Moodle LMS is a perfect example of an LMS that is great to use with iSpring-generated courses and is also free. We have just released a new version of iSpring Free, which upgrades it from Flash creating software to eLearning authoring master.

I’m not sure what upgrading to an “eLearning authoring master” means and haven’t taken the opportunity to test it out, so feel free to explore it on your own. Try this link (and let me know if it doesn’t work for you).


Ideas for Getting More HOW Into Your Face-to-Face Sessions

Trying to figure out how to make your sessions focused on “HOW” rather than “WHAT”? Though written for elearning development, the process and examples described in this article by Patti Shank can help you think through how to handle a session for FTF. This great Learning Solutions Magazine article, “The MOST Crucial Learning Activities and Media,” can be found here.


Flipped Classroom

Maybe you’ve tried this… it’s new to K-12 and catching on in higher ed.

You provide links to recorded lectures, discussions, and readings prior to the event, then focus on hands-on activities during the face-to-face (FTF) session. Though we should make it a goal to make this a model for educational sessions at conferences, established, stand-alone FTF events (institutes, seminars, etc.) are a good place to start. We often know more about our attendees, earlier, which is essential for making this model work.

I’d add that refusing to support poor learner behavior would also be critical to its success: if pre-event preparation is necessary for the FTF session to go well, you must resolve to start those sessions on time and with the assumption that learners have completed the preparatory assignments. No coddling those who “just didn’t have time” — doing so only punishes those who followed the rules and it jeopardizes your timeline.

So — keeping in mind that such a model will probably require managing some change around the current culture of the event — here’s more on the flipped classroom (or reverse instruction, as it’s also called). Note that these are just a few of the many resources a general Google search is likely to scare up for you:

“Flip your classroom through reverse instruction,” from The Electric Educator blog 

“Reverse Instruction: Dan Pink and Karl’s ‘Fisch Flip'” via the Connected Principals blog. 

“3 Keys to a Flipped Classroom” from the “David Truss: Pair-a-dimes for Your Thoughts” blog.

“The Revolution of Reverse Learning” from 2footgiraffe’s Blog.

“Advancing the Flip: Developments in Reverse Instruction” from the Connected Principals blog.


Why Knowing How the Brain Works Isn’t All You Need to Know

Yes, it’s important to know how we process new information and experiences, and that’s all about how the brain works. But don’t stop there. See “We are not our Brains,” from Norm Friesen’s blog  for more.


Need Coaching for Your Volunteer Online Content Leaders?

Looking for resources to provide guidance to your online volunteer content leaders? The University of New South Wales in Australia has provided some great tutorials, available free here. (Of course, don’t forget to access the free aLearning Fundamentals tutorial on “Leading Learning Events.” Just click the aLearning Fundamentals logo on the left.)


Help in Selecting an LMS

Janet Clarey’s “Wait. What? I can buy an LMS with a credit card?” doesn’t just provide a summary of her review of Intellum’s Rollbook LMS, but includes some great tips for your LMS/LCMS selection process. If you’re shopping for an LMS, you must read this post.


Ads on Twitter?

Mike at the electronic museum blog says it’s likely and cites the evidence he sees that your Twitter stream will soon carry advertisements in his post, “What if Twitter goes rogue?”

Don’t shrug this off. If you rely on Twitter, you need to think about what this could mean. Will you have any control over what those ads are for? What could it mean for your members, following a backchannel for your conference, to suddenly see an ad for something your organization doesn’t support? Ads on Twitter could mean a change you need to be ready to face.


Who Owns Your Photos In Social Media?

Kathy E. Gill at MEDIASHIFT asked this question, and answered it as well. Don’t assume you know the answer! (BTW… Facebook’s fine print on this matter is why I axed my account there.)


Attention K-12 Education Organizations!

If you or your members create online educational content or resources for students, teachers, administrators, or parents, you need to know about the “Learning Resources Framework Initiative” which will “improve search results for educational content on the Web” using a “common language of codes [that] Web producers and developers should embed within a digital learning object…”

Read about it in Education Week’s Digital Education column here.


Higher College Tuition Opens Opportunities for Associations

What could you be offering through your organization to those who might be affected by the rising cost of college tuition?

Here’s the data.

Because tuition and fees at public universities “have surged almost 130% over the last twenty years” while “middle class incomes have stagnated” fewer people are able to afford higher education. This creates a learning gap that your organization could fill… or some other organization could fill that gap, if you don’t. Why take that chance? So I’ll ask again: “What could you be offering through your organization to those who might be affected by the rising cost of college tuition?”


Social Media Tutorials

…Okay, they’re not tutorials in the sense that we use the term here at aLearning (the few links I clicked led to blog posts), but they’re still chockfull of great information, leads, and how-tos. Get started here, courtesy of Socialbrite.

Posted in aLearning Strategies, aLearning Trends, eLearning Resources, Learning in General, LMS, Measuring Results, Social Learning | Tagged: , , , , | 2 Comments »

Measuring Level Four

Posted by Ellen on June 24, 2011

Sounds like something from a sci-fi flick, doesn’t it?!? But of course I’m referring to Donald Kirkpatrick’s four-level model for measuring learning outcomes. The first level, you’ll remember, is “reaction.” We do a good job of measuring that by using “smile sheets” — those feedback forms that we issue right after learning as occurred (for more on Smile Sheets, see the article “Smile Sheets To Smile About” in the April 2010 issue of ASAE’s Associations Now magazine).

And whenever we “test” our learners on what they absorbed from a session, we’re measuring whether they learned (level Two on Kirkpatrick’s scale).

Levels Three (Behavior/Transfer) and Four (Impact/Results) are admittedly more difficult. They’re a challenge for corporations — and they have access to employee records, performance reviews, business outcome data, and all of that. How could we possibly begin to tackle these evaluative levels — and why would we want to try?

Let’s start with why. The answer is because.

Because we want our members to see evidence for themselves of the effectiveness of the training we’re delivering to them. The more we can demonstrate to them that they are benefitting (and their employers are likewise benefitting) from the educational sessions we provide, the more likely they are to renew their membership, register for more events, and tell others about the advantages they’re experiencing.

Because we want our association leaders to bear witness to the results of the programs we offer. Yes, they’ll see the attendance data, the revenue, and all of that, but showing them how members are contributing to their workplaces in ways they hadn’t before the training they took with us, the more powerful the rest of the numbers will be. This builds credibility for your department and should make it easier to gain their support for future program investments.

You can get insight into Level Three (behavior/transfer) by following up six to eighteen months after the event with an evaluation written specifically for this purpose (see “Nothing To Smile Sheet About”  and Chapter 17 of aLearning: A Trail Guide to Association eLearning for more on how to construct these evaluations).

But how do we get to Level Four? Much the same way we got to Level Three — by sending the session participants an evaluation that’s been carefully designed to solicit the sorts of responses that are experiencing the positive business impact we intended as a learning outcome.

Here’s how you might do that (adapt this to your own purposes, of course):

1. Get the learning objectives in front of you. If they were written well, they should provide the desired outcomes. For example, “The learner will be able to write effective broadcast e-mails that result in increased numbers of click-thrus.”

2. If your learning objectives weren’t written this clearly, brainstorm the possible business outcomes when the learning objectives are correctly applied.

3. Write questions that solicit specific business outcomes as a result of the session. Using our earlier example of broadcast e-mails, one question could be, “As a result of taking this training, have you experienced an increase in the number of click-thrus for your broadcast e-mails?”

4. Write follow-up questions that probe for details. For example, “What percentage of an increase in click-thrus have you experienced?”

5. Allow for exceptions — you can learn from these, too. For example, “If you haven’t experienced an increase in the number of broadcast e-mail click-thrus, describe the factors that could be affecting this result.” You might learn that they stopped sending broadcast e-mails or that someone else is now sending them and the learner doesn’t have the data. It could be that they always had a high rate of click-thrus so an increase that doesn’t seem significant is still a positive outcome.

Here are some examples of phrases to get you started:

“As a result of taking this training, have you experienced a decrease in…

…the cost of [X,Y,Z]?”

…employee turnover?”

…number of claiims?”

…number of errors in [A,B,C]?”

…number of complaints?”

…complaints about [A,B,C]?”

“As a result of taking this training, have you experienced an increase in…




…frequency of orders?”

…amount per order?”

…repeat business?”

…employee retention?”

…employee satisfaction?”

…customer satisfaction?”

…customer retention?”

“As a result of taking this training, have you experienced a savings in [X,Y,Z]?”

“As a result of taking this training, have you experienced enhanced creativity?”

“As a result of taking this training, have you reduced…




“As a result of taking this training, have you cultivated innovation?”

“As a result of taking this training, have you shortened your time to market with new products or services?”

“What other business outcomes have you experienced as a result of taking this training?”

Most importantly… after each question, ask for specifics:

How many? By how much? By what percentage did this change?

And of course you’ll want to emphasize that your data is strictly for evaluative purposes — you don’t need specific financial or other data, you just want some indication of the effect the training has had. Most members won’t release data that’s confidential to their company anyway, and some might be reluctant to even share that the training has made a business-side impact. That’s okay. Find out what you can from those who are able to share and consider yourself lucky to have that.

If the results are particularly stunning, follow up with individual respondents to see if you can use a quote from them for reports to the education committee, board of directors, or even in marketing materials. Offer to show them the quote and obtain their permission before releasing it. Being able to use specific testimonials is a plus — the real purpose of conducting this evaluation isn’t marketing, however.

When you have enough responses, aggregate the data so you can see the overall picture: how did learners benefit in general from the session? Was one objective particularly valuable? Was there any learning objective that seemed especially challenging? Why?

Thank every respondent, especially if their names are attached to the evaluations you get. Let them know how much you appreciate providing the feedback so you can continue to improve the program. A simple thank you goes a long way!

Has your organization conducted Level Four evaluations? How have you conducted them? What did you learn from the results? We’d love to hear your stories here at aLearning!

Posted in Learning in General, Measuring Results, Online Learning in General | Tagged: , , , , | Leave a Comment »