Outcome mapping

Outcome mapping is a project progress measurement system that was designed by the grant-making organisation International Development Research Centre (IDRC). It differs from traditional metrics in that it does not focus on measuring deliverables and its effects on primary beneficiaries but on behavioural change exhibited by secondary beneficiaries. The outcome mapping process consists of a lengthy design phase followed by a cyclical record-keeping phase. Outcome mapping is intended primarily for change focussed organizations that deal with complex systems and issues in changing environments, though it was originally designed for evaluating the impact of research in the developing world.

History

The outcome mapping manual was first published in 2001 by the Evaluation Unit of the IDRC. It is based on the outcome engineering model developed by Barry Kibel of the Pacific Institute for Research and Evaluation. Much of the terminology and procedures that occur in outcome engineering are also present in outcome mapping.

Overview of method

The key difference between outcome mapping and most other project evaluation systems is its approach to the problem that a project's direct influence over a community only lasts for as long as the project is running, and developing agencies have difficulty in attributing resultant change in those communities directly to the actions of the project itself. The outcome mapping approach is to focus less on the direct deliverables of the project and to focus more on the behavioural changes in peripheral parties affected by the project team. Thus an outcome mapped project report will focus less on the project's actual progress and more on the project's influence (both deliberate and unintended) during the project's progression.

Report-backs based on outcome mapping would tend to contain more anecdotal evidence than more traditional measurement approaches. Because of its deviation from traditional evaluation philosophy, outcome mapping may not be accepted by all donor organisations.

Outcome mapping consists of two phases, namely a design phase and a record-keeping phase. During the design phase, project leaders identify metrics in terms of which records will be kept.

Design phase

Writing two visionary essays

Project leaders begin the outcome map design by writing two short essays about what effect they would like to see their project have on the community and how they believe those influences would come about, written from a wishful thinking type of perspective, with the presupposition that their project is successful beyond their wildest dreams. The sole purpose of writing the essays is to facilitate a mind-shift among project leaders to the larger environment in which their project resides. In outcome mapping terminology, these essays are termed a vision and mission statement, but they have little in common with vision and mission statements used in strategic planning.

Identifying boundary partners

The philosophy of outcome mapping is that anyone the project influences is in fact a partner in that project. For this reason, project leaders draw a schematic (with the project in the centre) about which parties will be influenced by the project in any way (direct boundary partners), and which parties will in turn be influenced by those parties (indirect boundary partners). Project leaders select three or four boundary partners upon which they will focus additional activities (called primary boundary partners). Typically the direct recipients or beneficiaries of the project's deliverables will be one of the primary boundary partners. All further processes and reports within the outcome mapping with focus on various aspects of the primary boundary partners. In IDRC sponsored projects, project leaders are encouraged to include policy makers (e.g. governments) in their list of indirect boundary partners.

Defining progress markers

For each primary boundary partner, project leaders write a statement of desired overall behavioural change (called an outcome challenge) and a list of eight to eleven specific behavioural changes or actions the project would like the boundary partner to exhibit by the end of the project (called progress markers).

There are three types of progress markers, namely expect-to-see, like-to-see and love-to-see. These progress markers can be defined according to two approaches, namely term duration and degree of realism. In the term duration approach, expect-to-see are immediate responses that will be expected during the project itself, like-to-see are long-term responses that one would expect by the end of the project, and love-to-see are responses that one might expect after a number of months or years after the project came to an end. In the degree of realism approach, expect-to-see are items that are brutally realistic, like-to-see are items that are somewhat idealistic, and love-to-see are items that are actually unrealistic.

Identify tactics for boundary partners

Outcome mapping defines six types of actions that project leaders can take to effect each individual progress marker. These actions are filled in into a grid of tactics called in outcome mapping terminology a strategy map. The cells on the grid are called I1, I2, I3, E1, E2, and E3. I-class tactics ("internal" or "individual") are aimed at specific groups and individuals. E-class tactics ("external" or "environment") are aimed at the surrounding situation that groups or individuals may find themselves in.

Items I1 and E1 are called causal, I2 and E2 are called persuasive, and I3 and E3 are called supportive. Causal actions are directly related to the desired outcome and have a single purpose, persuasive actions are indirectly related to the desired outcome or attempts to produce the desired outcome indirectly, and supportive actions relate to providing and fostering an environment that enables or encourages the desired outcome with very little direct bearing on it.

It is not required to define actions for all six tactics for each progress marker—some progress markers may be suited to only one or two of these types of tactics. It is these tactics or actions that project leaders will perform during the project (in addition to producing the project's actual deliverables) and which project leaders will report back on. Interviews, with 6 consultants, occurred in order to learn experiences of the individuals in providing OM training. Together, the interviews with the summary of the data that was collected on applications and practitioners represent a valuable advance in comparison to the data previously held by OMLC. Of the status quo of OM, however, this study is not a good representation. User experiences and support available and required makes it distinctive for the experimenters and individuals.

Identify tactics for the visionary essays

After writing the visionary essays, and typically only after the rest of the design phase is completed, project leaders write down answers to eight specific questions about the visionary essays. These items are called organisational practices in outcome mapping terminology. The answers to these eight questions will be amended as the project progresses. The eight questions are:

1. Where or how will we find out about new opportunities or resources to achieve our vision and mission?
2. Who can we ask (in addition to our project beneficiaries) what our project beneficiaries think of our project?
3. How will we get support and approval from organisations' management or from government departments for what we do?
4. What procedures will be follow to ensure that we re-evaluate our methods continuously?
5. How will we determine from our boundary partners whether methods we intend to use, has been tried and found wanting?
6. How and what will we share about what we've learnt during the project?
7. What systems will we put in place to foster responsible experimentation with our limited resources?
8. When will we have feedback and bonding sessions for project management team members?

Record-keeping phase

In outcome mapping, three types of records can be kept, and it is largely up to the project leaders or donor organisation to decide which of the three (or all three) types of records should be reported back on. The records related to the items of the design phase.

Performance journal

The report named a performance journal is essentially a collection of minutes of the meetings at which the project's progress with regard to the organisational practices.

Strategy journal

The report named a strategy journal is a record of actions taken in terms of the strategy map (or tactics grid) along with results of such actions. The strategy journal is kept up to date continuously but may be required to be submitted to the donor at intervals.

Outcome journal

The report named an outcome journal is an anecdotal record of any events that related directly or indirectly to the progress markers (the expect-to-see, like-to-see and love-to-see items). The outcome journal is most useful towards the end of the project in providing the donor with visible impact stemming from the expenditure of funds, but may also be submitted to the donor at intervals.

Criticism

Since outcome mapping is more concerned about contribution than attribution, it is said that it should not replace traditional metrics but merely complement it. It has also been said that outcome mapping simply side-steps the attribution issue. An issue which has been addressed at IDRC workshops is the objection that outcome mapping procedures are too long and complex.

See also

• Evaluation
• Impact evaluation
• Logic model
• Logical framework approach
• Program evaluation
• Social return on investment

External links

• IDRC's official outcome mapping manual
• Outcomemapping.ca
• Two papers on outcome mapping at quality.bond.org.uk
• Slide overview of outcome mapping at evaluationinpractice.files.wordpress.com
• Barry Kibel on outcome engineering at ctassets.org
• Outcome Engineering Toolbox at pire.org