Synapsys Blog - November 2009

Gathering meaningful data for Return on Investment (ROI) analysis

One of the challenges in measuring return on investment in training/learning contexts is that much of the value can be intangible. Take for example training in new manufacturing processes. It’s relatively easy to measure ‘hard’ data such as reduced waste and rework. But what about intangibles that also come from the training.... ideas that improve the product, boosting sales.... staff who stay longer because the job is more rewarding.... It’s even harder when you move to measuring the impact of, say, a coaching programme on the performance of managers.

Our view? Be very clear about who will use the data, and what it will be used for.

Think about who the stakeholders are that will be looking at the results. What matters to them, and what evidence would they see as ‘valid’? Often stakeholders with a Human Resources/Learning and Development focus will be much more receptive to anecdotal data with more tenuous links to the bottom line....
• Three of ten managers all noted practical examples of improvement where they believe the cost saving to the company was greater than $2,000 so we can extrapolate that....
• The performance management cycle identifies that most trainees have improved faster this year in the area they got training in, so the programme is effective...

Those with a focus more on financial PKIs may discount this sort of ‘evidence’ as just speculation that cannot be meaningfully quantified.
• The manager’s perceptions are not reliable, and we can’t extrapolate over the whole company
• Improved performance is probably due more to the pay rises.... we can’t tell what the impact of the training was....

The problem is there is often a long chain of events between a training intervention and the company bottom line. If you require absolutely certainty before you recognise return, you’ll almost certainly underestimate the true ROI. Just because it’s hard to measure, doesn’t mean it’s not real.

So before you start, ask yourself:
1. Who will be relying on the ROI data?
2. What type of data will they need to see to have confidence in it?
3. What decisions are going to be made based on it?

Get this right before you decide on how you’ll measure ROI, and you’ll reduce the chances of your work being quietly binned.

Return on Investment (ROI) models

The starting point for most ROI models is the formula TB-TC/TC x 100. To work out your ROI, take the total benefit, deduct the total cost, and express the difference as a percentage of the total cost. Straightforward as far as it goes, but it doesn’t give you any clues on how to measure the costs and benefits…

There are a wide range of models that at least give pointers on how to approach the issue. Kirkpatrick’s four levels of assessing training is probably the most cited model. It suggests evaluation is conducted against four levels.
1. The learner’s reactions
2. What they have learned
3. How much that learning has been transferred into the work environment
4. What impact the transfer has had on the bottom line

You may be forgiven for thinking that the only measure that matters is level four. While it’s the ultimate measure of return on investment, evaluation at levels One to Three is critical if you aspire to continue and improve the intervention into the future.

Again, good as far as it goes, but how might you go about the measurement process? All effective models essentially include
1. Deciding on the impact you expect, and therefore what you need to measure
2. Measuring it
3. Analysing what you find
4. Reporting on ROI

Have a look at the Phillips Ten Step Model; it’s a good generic approach. (page 2).

We’ve been involved in several projects however where a valid ROI model was applied, and then ignored. Why? Because it didn’t measure the right things for the decisionmakers that mattered. Make sure your model gathers data that is meaningful to your organisation.