Self-study Module: Measuring Impact

When do you measure impact? 

You plan an evaluation and impact analysis at the start of your project or activity wherever possible. This way, you can place impact at the center of what you do. However, evaluation and impact are not the same. An impact-driven approach from the beginning of your project can help you and your team with project communication, audience engagement, and of course, setting up your data collection plans and workflows. The earlier you start, the more you gain.

Step 1: Determine who your audience is

Who do you want to reach with your activities? And for whom do you want to create and measure change? Your direct audience is the group for whom you bring about (positive) change directly. Your indirect audience may experience positive change because your activity influenced someone they care about or added value because your activity exists. It is challenging to measure change for everyone, so you will need to prioritize. Use the diagram on worksheet 2 for this.

Step 2: Consider the impact you want to make

As a team, view impact as the significant long-term goal that gives meaning and purpose to your work, as the driving force. For example, the mission of an organization for a more tolerant and inclusive society or a more robust local economy. Impact is something:

• That you cannot achieve alone
• For which you are not solely responsible
• That drives your organization or work (like the mission statement or vision)
• That cannot be realized tomorrow and may take a long time to achieve
• To which you may not be able to measure your direct contribution

It is now time to draft an impact statement. Continue with worksheet 3 and add your impact statement to the Change Pathway.

Step 3: Develop a plan

Outcomes are changes experienced by your audience, not activities you perform or results. So, develop a plan for the outcomes you want to create and the indicators you can measure. Consider what your audience might think, feel, or do. Organize a brainstorming session or group workshop and map out the short- and long-term outcomes for your audience. Don't forget the connections between these two types of outcomes.

Some things are always more important than others, and you cannot measure everything, nor should you try. Determine what you want to achieve and then identify which changes experienced by your audience are the most important to measure.

Once you have identified the most important outcomes, it's time to think about how to measure them. Indicators are the benchmarks that help you determine whether a particular outcome has occurred.

Indicators can be subjective (a self-reported experience) or objective (an objectively obtained measurement). A combination of these two is a good idea if you want to determine whether change has occurred.

The final step in your Change Pathway is to map out your outputs. Outputs allow you to quantify your project.

Use worksheet 4 to map out the above components.

Step 4: Collect your data

Create a data collection plan

Your data collectively form the information you need to answer your research questions and conduct your analysis. To create a data collection plan, you need to determine:

• From whom you want to collect data?
  
• Where and when?
• How, using which methods?
• How much data you need to collect to get a good and reliable picture of your audience's experiences?
• How you can integrate data collection into your existing workflows?
  

Decide which research methods you can use

Depending on what you want to measure, you can collect data using many different methods. Quantitative data are data you can count, while qualitative data are stories, opinions, etc. Ensure that whatever data collection method you choose, you obtain the type of data you need. Learn more about research methods.

Whenever possible, collect baseline and longitudinal measurements

To measure the impact of something, it is useful to try to conduct a baseline measurement that you can compare with a later measurement. This indicates that change has occurred. For the long term, it is a good idea to measure change not only immediately after an activity but also some time later. This is longitudinal data collection.

Step 5: Data analysis and interpretation

Start by describing what you see in your data. What can you infer from the data? You can analyze your data using data analysis tools. For qualitative data, it is important to bundle the themes, connections, and key information in your data. You should ask yourself questions such as:

• What do the data tell us about the research questions, indicators, and outcomes we identified?

• What does this mean for what we already knew or assumed?

• What information do the data not provide, and what do we still not know?

• Does this sound plausible, and why?

• Is there anything unexpected?

Step 6: Write your impact story

Ensure that your data lead to impact for you! It is now time to write and visualize your impact story.

The presentation of your findings in charts and graphs is just the beginning of your impact story. An impact story is based on your analysis and interpretation of your data. In your story, you should not only clearly articulate what you have learned but also why that is important and what you will do with those lessons. This is a creative process where you need to think about who you share this information with and what you want to achieve.

This also applies to data visualization. It's not just about presenting the data but also interpreting what the charts and graphs show and why that is important. Remember that less positive results are just as important in your impact story, just like the plot twist in a traditional story.

Use worksheet 5 for help with how to:

• Sketch the initial situation using your objectives and impact goal
• Introduce your audience as characters in your story
• Outline the plot based on the key findings
• Introduce your data objectively through plot twists
• Reflect on what it all means.

Next steps

In the preparation for this module, we asked you what you wanted to achieve with an impact analysis or evaluation. Did you want to improve your digital offering for your audience? Did you want to communicate your impact more effectively to your funders for more sustainable funding? Did you want to increase the budget for digital culture at the school you collaborate with? Now you know what you want, and it's time to turn these intentions into actions. 

Even more impact

Use your insights and experiences in a future activity. Ensure that impact thinking becomes part of everything you do. Curious about how others do this? Read the practical examples of DutchCulture and Festival Cement.