Synapsys Blog - Evaluate

Gathering meaningful data for Return on Investment (ROI) analysis

One of the challenges in measuring return on investment in training/learning contexts is that much of the value can be intangible. Take for example training in new manufacturing processes. It’s relatively easy to measure ‘hard’ data such as reduced waste and rework. But what about intangibles that also come from the training.... ideas that improve the product, boosting sales.... staff who stay longer because the job is more rewarding.... It’s even harder when you move to measuring the impact of, say, a coaching programme on the performance of managers.

Our view? Be very clear about who will use the data, and what it will be used for.

Think about who the stakeholders are that will be looking at the results. What matters to them, and what evidence would they see as ‘valid’? Often stakeholders with a Human Resources/Learning and Development focus will be much more receptive to anecdotal data with more tenuous links to the bottom line....
• Three of ten managers all noted practical examples of improvement where they believe the cost saving to the company was greater than $2,000 so we can extrapolate that....
• The performance management cycle identifies that most trainees have improved faster this year in the area they got training in, so the programme is effective...

Those with a focus more on financial PKIs may discount this sort of ‘evidence’ as just speculation that cannot be meaningfully quantified.
• The manager’s perceptions are not reliable, and we can’t extrapolate over the whole company
• Improved performance is probably due more to the pay rises.... we can’t tell what the impact of the training was....

The problem is there is often a long chain of events between a training intervention and the company bottom line. If you require absolutely certainty before you recognise return, you’ll almost certainly underestimate the true ROI. Just because it’s hard to measure, doesn’t mean it’s not real.

So before you start, ask yourself:
1. Who will be relying on the ROI data?
2. What type of data will they need to see to have confidence in it?
3. What decisions are going to be made based on it?

Get this right before you decide on how you’ll measure ROI, and you’ll reduce the chances of your work being quietly binned.

Return on Investment (ROI) models

The starting point for most ROI models is the formula TB-TC/TC x 100. To work out your ROI, take the total benefit, deduct the total cost, and express the difference as a percentage of the total cost. Straightforward as far as it goes, but it doesn’t give you any clues on how to measure the costs and benefits…

There are a wide range of models that at least give pointers on how to approach the issue. Kirkpatrick’s four levels of assessing training is probably the most cited model. It suggests evaluation is conducted against four levels.
1. The learner’s reactions
2. What they have learned
3. How much that learning has been transferred into the work environment
4. What impact the transfer has had on the bottom line

You may be forgiven for thinking that the only measure that matters is level four. While it’s the ultimate measure of return on investment, evaluation at levels One to Three is critical if you aspire to continue and improve the intervention into the future.

Again, good as far as it goes, but how might you go about the measurement process? All effective models essentially include
1. Deciding on the impact you expect, and therefore what you need to measure
2. Measuring it
3. Analysing what you find
4. Reporting on ROI

Have a look at the Phillips Ten Step Model; it’s a good generic approach. (page 2).

We’ve been involved in several projects however where a valid ROI model was applied, and then ignored. Why? Because it didn’t measure the right things for the decisionmakers that mattered. Make sure your model gathers data that is meaningful to your organisation.

Facts and Figures in Adult Learning

Here are some interesting figures (many U.S. based) for how adult learning is faring and meant to grow over the next few years.

Link here...

Blended Learning and Business Change

This is a semi-introductory article on blended learning, but includes interesting results of an IDC study of fifteen organisations that have undertaken blended learning. Included are some helpful guidelines for developing a blended learning approach, focused on ensuring business impact.

http://www.clomedia.com/content/templates/clo_fairfield.asp?articleid=349

Synapsys Presentation on Training Effectiveness

We delivered this presentation on Dec. 11th as the first in our series of eLearning solutions seminars. In this one, we discuss what effective training means and how to measure effectiveness, including the pros and cons of quantitative vs. qualitative approaches to measurement.

Click here to launch the presentation in your browser...

Aligning Training with Business Strategy

This is a subject near and dear to our hearts and one that is now getting a lot of attention: the importance of aligning training to business strategy and looking for real changes in behaviour and improvements to the bottom line. This article is a discussion about ROI to some extent, but its strongest message is the importance of executive involvement in training decisions.

http://www.optimizemag.com/issue/026/roi.htm

Research on Current Practice

In May 2001 Synapsys participated in a survey on online learning practices. The results of that survey are now available at http://www.publicationshare.com/docs/corp_survey.pdf

Respondents were from the corporate sector, spanning all sizes and industries, the largest respondents being education, I.T., Finance and Industrial. The results provide an interesting snapshot of how things were at that time, and canvas a range of issues including:

-barriers to learning
-organisational structure and training methods
-development methods and pedagogical approaches
-communities and resources
-methods for engaging learners
-Return on investment (ROI) measurement, or lack of it
-emerging technologies

Bear in mind the snapshot is of two years ago, but it still makes for interesting reading. There are 15 broad recommendations covering areas such as:

-learning objects
-online learning policies and procedures
-tool development
-freelance instructors/designers
-information portlas and communities

Real-life Lessons About eLearning

This is a fascinating summary of a talk given by Damien Faughan, Charles Schwab's Director of Infrastructure & Technology, at the eLearning Producer conference. Damien frankly discussed lessons the large financial services company has learned about eLearning over the last several years, including issues related to technology, reusability, learner motivation, blended learning, etc.

These lessons learned really substantiate many of the principles we espouse at Synapsys, for instance: "At the end of the day, all learning needs to be strategic and transformational, learner-centered and focused on contributing to the business."

http://www.internettime.com/blog/archives/001021.html

Blended Learning Boosts Productivity

A new study by NetG shows that blended learning can offer a substantial improvement on productivity over methods that employ just one delivery option:

http://www.learningcircuits.org/2003/mar2003/newsbytes.html#1