Skip to main content

​
Measuring Impact beyond ben num

Measuring Impact: Beyond Beneficiary Numbers Headline statistics are a tricky thing; we use them to communicate impact but we also harbour doubts about representativeness.  When it comes to beneficiary numbers, the key, we’ve learned, is to picture beneficiaries on a scale of intensity. An example from the Criminal Justice sector In my previous job, we operated in Court Custody suites and conducted around 300 daily paper-based ‘screenings’ of defendants’ information which might indicate mental health or social needs. This might lead to 30 full mental health assessments out of those individuals screened. 3 of those might result in a recommendation for a full assessment under the Mental Health Act. Let’s do some maths: 300 + 30 + 3 = 333 beneficiaries - right? Keep up at the back! There’s a very different amount of work in each case, making this a pretty meaningless calculation. In this example, there are 300 ‘low-touch’ beneficiaries whose records were screened. Some of these might become more of a ‘medium touch’ piece of work, e.g if you have a face to face interaction with the individual, and make a call to their GP/ other services. The 30full assessments are almost certainly ‘high touch’ and ‘hands-on’ work getting to know an individual and their needs. When you talk to a practitioner in the sector, these categories will make some sense to them and instinctively they will have an idea of how this applies to different service users they’ve seen. CAN Beneficiary Scale How does this apply back to us? In our end of award survey to entrepreneurs, we have always asked how many ‘unique’ individuals benefited from their work. But we recently challenged ourselves on the utility of this figure, which is without context and hard to accurately obtain or verify. We started to trial instead the CAN Invest beneficiary scale, which encourages us to think within the low/medium/high touch classification and to unpack the following dimensions of the interaction with a beneficiary: How many interactions an entrepreneur has with an individual How often do these interactions take place? How long do these interactions last? What is the most common medium for the interaction? Methodology: Using the Scale Step 1: Collect data from entrepreneurs in accordance with the above categories Step 2: Use the CAN Beneficiary ‘Intensity’ formula (below) to produce an intensity score for each entrepreneur we support: Step 3: Use industry benchmarks to classify each entrepreneurs beneficiary model as low, mediumor** high touch** based on statistical quartiles.  Step 4: Work out how to use the results! Results Running this calculation on 134 award winners who answered our most recent survey suggests several things: Almost half the entrepreneurs surveyed worked at a high touch level with beneficiaries. That’s an** estimated 35,000 individuals benefiting in a high touch way from that sample alone**.  Entrepreneurs who work in a low-touch way tend to have a greater number of beneficiaries Our Total and Mean beneficiary numbers are being inflated by outliers at the low touch level. (Think ventures with ‘abnormally’ high beneficiary estimates) Moving Forward We’re now starting to think about how we might use this data to more confidently assert the impact of our entrepreneurs. But we’d like to hear any thoughts you might have. For example: How does the data presented above match our sense of ‘what we thought we knew’ about entrepreneur impact? Can you think of ways in which data like this supports what you may be trying to achieve? Does the** low/medium/high touch** distinction help with your thinking about impact? In many ways what we’ve done so far has been the easiest part; it’s much harder to decide how you’re going to use your data proactively and which audiences will benefit from it. In research, we want to be brave and bold in communicating our impact, but also accountable and clear about the assumptions made in reaching these numbers. We want to celebrate our entrepreneurs but also practise ‘responsible statistics’. Watch out for future updates as we try and find this happy middle way in 2018!

​
S4AS CIF

​
Introduction

UnLtd partnered with the University of Northampton’s Institute for Social Impact and Innovation (ISII) to develop a Collective Impact Framework (CIF) for our Solutions for an Aging Society (S4AS) impact area. The research conducted 10 interviews (7 UnLtd staff and 3 socents) to identify the priority ‘impact sectors’ to generate social impact measures for UnLtd and our socents to measure impact. The research also drew on ISII’s sector knowledge and UnLtd’s S4AS Theory of Change to generate the following list of priority impact sectors for the CIF: This briefing paper will cover the following topics:
  • Review of UnLtd’s S4AS overall impact goal and the alterations made based on the CIF research;
  • Overview of the CIF and the ‘pick ‘n’ mix methodology approach to impact measurement selection;
  • A set of next steps within UnLtd to embed the CIFs and contribute to UnLtd’s goal of evidencing the impact of our strategy.

​
Special note on the S4AS Goal

As part of the CIF research, ISII investigated UnLtd’s S4AS goal to improve healthy life expectancy and measuring this impact using the Hale Average Life Expectancy (HALE) measurement approach. This is demonstrated in Figure 1 below: Figure 1: S4AS UnLtd Theory of Change ISII’s research found HALE to be a complex research approach that is not appropriate for UnLtd and the socents we work with. Secondly, the research found that the majority of UnLtd’s socents do not work directly in affecting health outcomes *before *healthy life has deteriorated. Instead, many of UnLtd’s social ventures focus on ensuring a better (or prolonged) quality of life for older people once healthy life deteriorates. Therefore, the impact goal will shift as follows: UnLtd’s primary goal for the S4AS Impact Area will be to Increase the Quality of Life for older people in the UK. This will be measured through the BBQ Quality of Life scale (see next section for details) and can also be measured by proxy through other scales focusing on Life Satisfaction, Social Connection and Wellbeing. All future UnLtd communications, outreach and delivery should consider how socents contribute to the overall impact goal of improving quality of life for older people in the UK. This does not exclude health focused interventions as health is a key component of having a good quality of life. However, the view of health will be as contributing to a ‘holistic’ approach to a quality life.

​
The Collective Impact Framework & impact reporting

The CIFs provide a framework of quality assured metrics for measuring social impact. Currently the CIFs are held in the Microsoft team folder . The aim of the CIFs is to provide a pick ‘n’ mix menu of measurement approaches based on the socent and the type of impact they deliver. Therefore, each impact sector has a ‘bucket’ of ways to measure outputs, outcomes and impact. This can be demonstrated below: The CIFs will be used in two primary functions:
  • For our Award Winners and Ventures: A framework of quality assured metrics to access to start and improve their social impact management (SIM). As will be discussed later, UnLtd also propose to set up a ‘data exchange’ for our social entrepreneurs to submit their social impact for support on analysing and reporting social impact.
  • For UnLtd: The CIFs will provide a step towards standardised and comparable reporting from our award winners and ventures. This will enable us to better advocate for social entrepreneurs with evidence of their impact and to build a bank of data and insights to evidence our progress towards achieving our strategic impacts.
  • For our Award Winners and Ventures It is critical to ensure our social entrepreneurs make the most of accessing the CIF. UnLtd must recognise the where SIM fits in with an entrepreneur’s priorities. Therefore, the following ‘levels of evidence’ is recommended for our social entrepreneurs to establish before the are introduced to more thorough SIM through the CIF.
UnLtd’s Research, Impact and Learning (RIL) team will work with AMs and VMs to set the criteria of existing KPI (Key Performance Indicator) requirements to be in place before a social entrepreneur starts collecting data against the CIF. Once a social entrepreneur has been identified as ‘ready for CIF’ the AM/VM will work with RIL to design a short support package to establish what data exists or needs collecting and to agree a ‘data exchange’ point.
  • For UnLtd UnLtd’s strategy outlines a clear requirement to understand how we work with socents in the S4AS sector. To do this, UnLtd must draw on multiple data sources to understand:
  • Our reach: The number of socents we work with and the number of older people they work with. This will help us understand what our role is within supporting the market to improve quality of life for older people.
  • The type of interventions we support: We need to understand how our socents work with older people and the interventions they offer. If 90% of our socents deliver physical health training in care homes, then we will probably not see an impact on social care in the community. Therefore, we need to categorise the ‘impact sectors’ our socents fall into.
  • The Social Value generated by socents using the CIF: Built up short case studies using CIF measurements to understand the social impact of various business models. As per the section on benefits to socents, this will be known as the ‘data exchange’.
This has been demonstrated below as the levels of data UnLtd will need to demonstrate our social impact and to plan the rollout of our CIF work:

​
What we can say and when we can say it

How we talk about our impact is critical for ensuring our social impact reporting is taken seriously. The CIF will provide a framework to compare and aggregate data to estimate the net social impact of our work. This will be built up over time and **the more buy in and support from ****frontline staff to promote excellent SIM with socents will be critical to the success of UnLtd understanding and demonstrating our impact in S4AS. ** As we build up our evidence base, RIL propose the following communication proposals for our social impact work:

​
Next steps

We have only started our CIF journey and will constantly test ways of making the CIFs as user friendly as possible for our socents and for UnLtd staff. The following key next steps within 2019/2020 will be taken in the short term to begin rollouts of the CIFs with internal teams and to begin embedding in new programmes: **Business Development **
  • Work with the S4AS II design team to ensure the ‘data exchange’ and requirements of CIF reporting are effectively integrated into the S4AS II programme. **Ventures **
  • Work with the Transform Aging team’s final evaluation to map the Transform Aging ventures into the impact sectors in the CIFs.
  • Testing the roll out with Thrive Cohort 2 S4AS socents including:
  • Venture Selection
  • Support planning
  • Support delivery
  • Begin conducting case studies ‘data exchange’ services through UnLtd’s RIL team working with identified ventures. **Awards team **
  • Internal training on the CIFs for AMs/VMs and any other relevant staff.
  • Work with AMs to design how AMs can ensure effective tagging and identify how the rollout of impact sector tagging and socent CIF readiness can be identified.
  • Engage with our newly appointed SEIR to design S4AS-specific pathways for training socents to have basic data in place to benefit from the CIF.
Application Process
  • Review of the existing social impact tagging and identification to better improve alignment to the CIFs. **Reach and Influence **
  • Work with R&I team to ensure all new communications focus on quality of life within the S4AS impact area rather than healthy aging.
SIM element2019/20202020/20212021/20222022/2023
Social Impact achievedWe have 5 – 10 case studies of the social impact generated by our social venturesWe have a core bank (>20) of case studies of social impact using the CIF and can talk about the aggregate impact of these confidentlyUsing prudent extrapolation, we can estimate the aggregate social impact of our S4AS socents based on our core bank of case studies and the CIF to inform comparisons of impact.In S4AS, we have worked with XX social ventures to generate the following aggregate social impact.
Types of interventionWithin our ventures programme, our socents work on the following key impact sectors: wellbeing 20%, social inclusion 40%, physical health 20% etc.Across our portfolio, we have tested what impact sectors our ventures and some award winners work in. The distribution is as follows….Across our whole portfolio, our socents are split across impact sectors as: 40% on quality of life, 20% on wellbeing etc. etc.Across our whole portfolio, our socents are split across impact sectors as: 40% on quality of life, 20% on wellbeing etc. etc.
Reach of S4AS socentsWe have worked with XX socents in S4AS and they have worked with XX beneficiaries.We have worked with XX socents in S4AS and they have worked with XX beneficiaries.We have worked with XX socents in S4AS and they have worked with XX beneficiaries.We have worked with XX socents in S4AS and they have worked with XX beneficiaries.

​
Downloadable Resources