Elementum
← Back to impact measurement and evaluation

Collect and Make Honest Sense of the Evidence.

You are gathering data, but it is not telling you anything, or you are afraid of what it might tell you.

Data that is collected but never honestly interpreted is worse than none, because it costs effort and buys no truth, and sometimes it quietly buys a comfortable lie. This work gets the data collected well and, harder, makes honest sense of it: separating what changed from what you merely did, facing the answer squarely when it is not what you hoped, asking honestly whether the change is really yours, and reaching conclusions you can actually stand behind. It is the honest heart of impact measurement: the willingness to look at the evidence and believe it. It is not statistics for their own sake, and it does not require you to be a researcher. Use this if you have a measurement system and are collecting data, or ready to, but cannot make honest sense of what it says. If you do not yet have indicators and a way to collect them, go to Build the Measurement System first. And come with the one thing it truly requires, which is not a skill but a stance: the willingness to believe the evidence even when it disappoints you.

Step 1

Collect the data well and with consent

Honest conclusions require honestly gathered data: collected consistently, from the sources you planned, with the consent and care the people you serve are owed. Run the collection routine as designed, with proper consent, protecting privacy and dignity, and gathering consistently rather than scrambling before a deadline. Then check for the gaps and biases that would mislead you: who is missing from the data, whose voices dropped out, where collection was patchy.

Who: you or your program director, with whoever owns your data, who is accountable for its quality and its protection. Produces: a body of evidence collected consistently and with consent, with its gaps and biases named.

If collecting surfaces a disclosure of harm, abuse, or danger

That is not a data point to record and analyze; it is a duty to act at once, and it routes immediately to the relevant professional, the safeguarding process, and where required an attorney or the authorities. The person safety comes before the evaluation. Where anyone is at risk, this takes precedence over the rest of this work.

Open the Collection Quality and Consent Checklist →
Step 2

Separate what changed from what you merely did

Analysis here is not complex statistics; it is the disciplined separation of outcome from activity, and the honest look at who changed, by how much, and for whom. Work through the data to see what actually changed for the people you serve, against your indicators, rather than how much you did, looking at who changed and who did not. Then look for the honest patterns: where the program works well, where it does not, and for whom, because an average can hide that you help one group and fail another.

Who: you or your program director, with the data owner. Produces: an honest reading of what actually changed, for whom, and by how much, with the patterns an average would hide brought into the open.

Open the Outcome Analysis Worksheet →
Step 3

Reach the honest read, including the answer you did not want

This is the step the whole work is built to reach and the one an organization is most tempted to skip: looking at the evidence and saying, honestly, what it means about whether the work works, and whether the change is really yours. Say plainly what it shows, including if the answer is that the work is not producing the change, or not as much as you believed, and do not explain it away. For each change you see, ask whether your work plausibly caused it or whether other things in the person life could explain it, and scope your claim to what you can honestly defend.

Who: you or your program director, with a board member and an honest outside reader. This read cannot be delegated to the person with the most to lose from it; where the finding is hard, it is a leadership and integrity act, and the board should see it. Produces: an honest reading of what your evidence shows, including unwelcome findings, with each claim scoped to what you can genuinely attribute.

If a real decision rests on airtight proof of cause

If a major funder or a public claim genuinely rests on airtight proof that your program, and not something else, caused the change, honest self-service measurement has reached its limit, and that proof is the work of a professional evaluator, the way a hard legal question is the work of an attorney. Name what you can honestly claim from your own evidence, and route the airtight-causation question to a professional evaluator. This is a route to a specialist, not a stop, and it is a boundary of honesty, not a lack of capability.

Open the Honest Read Instrument →
Step 4

Draw conclusions you can stand behind

The honest read becomes useful only when it is turned into clear conclusions: what you now know, what you still do not, and what it means for the work. Write the conclusions you can stand behind: what the evidence shows, how confident you are, what remains uncertain, and what it means for the program, as clear about the limits as about the findings. Then name what these conclusions call for, whether that is changing the program, gathering better evidence, or reporting a real result.

Who: you or your program director, with the data owner and a board member. Produces: a clear, defensible set of conclusions about your impact, honest about both findings and limits, that becomes the basis for improving the work and reporting it truthfully.

Open the Evidence Conclusions Record →

The honest edge

Two edges meet here. The first is the limit of self-service rigor: honest, practical measurement can tell you a great deal, but it cannot, on its own, prove beyond challenge that your program alone caused a change, and where a high-stakes decision or a demanding funder requires that proof, a professional evaluator is the honest answer, named as a boundary and never as a substitute for the work you can do yourself. The second is the duty that overrides the evaluation entirely: if your data collection surfaces harm or abuse or danger to a person, that is not a finding to analyze but a duty to act on at once, routed to the right professional and, where required, an attorney or the authorities.

How you will know it worked

You can say what your evidence shows about your impact in plain, honest terms, including where it disappoints you, and you have not explained the disappointment away. You can tell where the program works and for whom, rather than hiding behind an average. You know which of your claims you could defend to a skeptic and which you could not. And your board has seen the real read, not a curated one.

Where to go next

With the evidence understood, your plan most likely sends you next to Use and Report the Evidence. Where a decision needs airtight proof of cause, that routes to a professional evaluator; where collection surfaced harm, that routes at once to the right professional and, where required, an attorney or the authorities.

You can always go back to the overview or start over from the welcome page.