The Evidence Check: a programme
Route: A programme Choose a different route
JavaScript is off, so here is the whole check as a checklist you can print or fill in on paper. A key to the results is at the end.
A programme
-
1. What’s the main thing riding on it in the next 12 months? About you, not scored
-
2. Can you say in one sentence what should be different for participants afterwards? Outcomes
-
3. What is the strongest evidence you currently collect? Evidence of change
-
4. When do you measure outcomes? Evidence of change
Answer only if your answer to question 3 is a knowledge test or anything stronger, or Not sure.
-
5. Where do your outcome questions come from? Measure quality
Answer only if your answer to question 3 is a knowledge test or anything stronger, or Not sure.
-
6. If outcomes improved, could you rule out other explanations? Attribution
Answer only if your answer to question 3 is a knowledge test or anything stronger, or Not sure.
-
7. Do you check the programme was delivered as designed? Delivery
Answer only if the programme has run.
-
7. Will you check it’s delivered as designed? Delivery
Answer instead of the question above if the programme hasn’t run yet.
-
8. What happened the last time evidence came in? Decision use
Answer only if the programme has run.
A measure
-
1. What is it mainly used for? About you, not scored
-
2. What’s the main thing riding on it in the next 12 months? About you, not scored
-
3. How was the content of the measure decided? Content
-
4. Has it been checked (piloted or validated) with people like the ones you use it with? Content
-
5. Is there evidence that scores relate to what they should, such as performance, later outcomes or other measures of the same thing? Validity
-
6. If two scores differ by a few points, could you say whether that’s real or noise? Precision
-
7. Has anyone checked it works the same way across groups, such as sex, ethnicity, age or first language? Fairness
-
8. Do you use cut-offs or score bands? Decisions
-
8b. How was your cut-off or band, such as ‘high potential’ or ‘at risk’, set? Decisions
Answer only if you answered Yes to question 8.
-
9. Has it been shown to pick up real change? Change
Answer only if you use it for measuring change over time.
-
10. If a client, auditor or regulator asked for the technical evidence tomorrow, what would you send? Evidence ready
Reading your answers
- Each scored question belongs to an area, named beside it. An area’s result: Not sure if you answered Not sure to any of its questions; Not applicable yet (programme) or Not applicable (measure) if you skipped all of its questions (on the programme route, Not yet reached instead if you skipped them because your answer to question 3 was attendance or satisfaction: Once you measure outcomes, you’ll also need a way to show the programme caused the change.); In place if every answer scored 3 points; Not described if every answer scored 0 or 1; otherwise Partly in place.
- Where to start: any area that is Not sure or Not described, then any that is Partly in place. Where two areas are level, take them in this order. Programme: Outcomes, Evidence of change, Measure quality, Delivery, Attribution, Decision use. Measure, by what it is mainly used for (question 1): Hiring or promotion: Validity, Fairness, Decisions, Content, Change, Precision, Evidence ready; Development or coaching: Content, Validity, Fairness, Decisions, Change, Precision, Evidence ready; Screening or identifying need: Decisions, Content, Validity, Fairness, Change, Precision, Evidence ready; Measuring change over time: Change, Content, Validity, Fairness, Decisions, Precision, Evidence ready.
A useful first step, by area
- Programme: Outcomes
- Write one sentence: who will do what differently, by when. Agree it with whoever signs off the budget.
- Programme: Evidence of change
- Ask participants again 6 to 8 weeks later what they have done differently, not how they rated the course.
- Programme: Measure quality
- Keep outcome questions word-for-word identical at every time point, and check whether they have ever been tested for detecting change.
- Programme: Attribution
- Collect a simple baseline before your next group starts.
- Programme: Delivery
- Give trainers a short after-session checklist: what was covered and what was skipped. If the programme hasn’t run yet: Before launch, agree what ‘delivered as designed’ means (the core sessions, content and contact time) and how trainers will record it.
- Programme: Decision use
- Before collecting anything, write down the decision the evidence will inform and who makes it.
- Measure: Content
- List what the measure claims to capture and map every question to one claim. Gaps and orphans show up quickly.
- Measure: Validity
- Pick one outcome the scores should predict or relate to, and check whether the two line up in data you already have.
- Measure: Precision
- Find your reliability figure and ask what margin of error it implies for one person’s score.
- Measure: Fairness
- Compare responses question by question across the groups you have data for, not just average scores.
- Measure: Decisions
- Write down where each cut-off came from. If you can’t, start there.
- Measure: Change
- Check the measure’s published evidence for detecting change before your next before-and-after.
- Measure: Evidence ready
- Gather what you have into one folder with a one-page summary.
Why it matters, by what’s riding on it
- A renewal, tender or sale
- This is the question a buyer comparing providers is likely to ask.
- A funder or board report
- This is the gap a funder or board is most likely to probe.
- A regulator, auditor or client’s due diligence
- This is where external scrutiny usually starts.
- Mainly our own improvement
- Fixing this first makes every later evaluation cheaper and more useful.
- Nothing in particular
- Fixing this first makes every later evaluation cheaper and more useful.
What your evidence can and can’t say (programme route)
Your answer to question 3, the strongest evidence you collect, puts you on one rung of this ladder.
- Rung 1, Took part. Can say: people attended. Can’t yet say: they learned or changed anything.
- Rung 2, Valued it. Can say: people rated it well. Can’t yet say: anything changed.
- Rung 3, Learned it. Can say: people knew more or felt more confident at the end. Can’t yet say: it lasted or changed how they work.
- Rung 4, Can do it. Can say: people can apply it in a realistic task. Can’t yet say: they do it at work.
- Rung 5, Do it at work. Can say: how people work has changed. Can’t yet say, without a comparison: the programme caused it. If you have a comparison (question 6): Can say, with your comparison: the programme likely contributed. If you measure straight after only (question 4): Can say: how people work now (without a before measure, not that it changed).
- Rung 6, Results moved. Can say: team or organisational results improved. Can’t yet say, without a comparison: it was the programme. If you have a comparison (question 6): Can say, with your comparison: the programme likely contributed. If you measure straight after only (question 4): Can say: team or organisational results now (without a before measure, not that they improved).
Your next step
A programme: Want to know what your existing evidence can support? An Evaluation Framework Review looks at what you already collect and what a stronger evaluation would need. A free 30-minute call is enough to see whether it fits.
If the programme hasn’t run yet: Starting from scratch? An Evaluation Framework Review gives you an evaluation plan you can commission against. A free 30-minute call is enough to scope it.
A measure: Want a clear verdict on your scores? An Assessment Health Check uses the response data you already have. A free 30-minute call is enough to see what it needs.
Or email your results to greg@gregorygorman.co.uk.
An initial guide based on your answers, not an independent review or a validated assessment. Nothing you entered was stored or sent.