Understanding risk and success in social impact bonds

ThinkForward’s Social Impact Bond was one of the first in the world to successfully complete. Thanks to this investment we not only transforming the lives of over a thousand of the most disengaged young people in London but investors in ThinkForward’s programme also reaped a financial return.

Since completing the bond we have identified some key lessons highlighted in Philanthropy Impact’s December 2016 issue. To view the full article click here or read the below.

In the UK and continental Europe, SIBs are being used primarily to experiment with new interventions, such as the programmes commissioned by the UK’s Department for Work and Pensions Innovation Fund to support disadvantaged young people. In the US, however, ‘pay for success’ initiatives are generally used to scale existing evidence-based programmes. Following the launch of the first SIB in the UK in 2010, much was written to describe the mechanics, structures and challenges of implementing new deals. Over 40 SIBs have been launched since then – in the UK, US, Australia, Canada, Netherlands, Belgium, Portugal and Germany. As investors crowd-in and the first round of SIBs have been completed, the buzz is less about how to design them and more about whether they have worked. What are the lessons from the first SIB deals? What should investors and advisors know about measuring success and risks within SIBs? As investors crowd-in and the first round of SIBs have been completed, the buzz is less about how to design them and more about whether they have worked. Based on our experience developing SIBs, we have identified some key lessons. Firstly, investors should be aware of the risk that SIBs may not reach completion as planned. The first SIB in the world was launched in the UK in 2010 with the One Service in Peterborough Prison. It was designed to fill a gap in the UK criminal justice system by supporting short-term prisoners and preventing their reoffending. Social investors invested £5m in a project meant to run for seven years. But in 2014 the Ministry of Justice announced a decision to restructure the provision of probation services nationally and the Peterborough pilot was brought to an early close.

The New York Times (Wall St. Money Meets Social Policy at Rikers Island) reported a similar outcome for the first SIB in the US, launched in 2012 to support high-risk adolescents detained at Rikers Island, the New York City jail. In this case, it wasn’t the change of heart of policymakers but implementation issues: the control group fell apart as wardens ‘could not separate teenagers who were to participate in a course of cognitive behavioural therapy from those who were not supposed to attend’, and then there was the failure to recruit the teachers required to carry out the therapeutic programme after the city’s Education Department pulled out. Was this a total failure? Investors will certainly be disappointed: Goldman Sachs had invested $9.6m with a 75% guarantee from Bloomberg Philanthropies. But from the perspective of the City of New York, it wasn’t a disaster: the city had tested a new intervention at scale with the benefit of a rigorous evaluation and then walked away without spending any taxpayer money. So another key lesson for investors is the importance of focusing part of their due diligence around their partner’s record of, and ability to, recruit beneficiaries and frontline workers. At ThinkForward SIB, our programme in London helps more than 1,000 at risk young people aged 14 to 19 years to complete their education and transition into work. The challenge of recruiting beneficiaries was partially solved by working inside schools. ThinkForward SIB carefully selected partner schools where teachers were committed to refer, and give access, to students. And when it came to staff, ThinkForward investors took a conscious decision to pay a competitive salary to frontline workers in order to secure the most talented applicants. When SIBs reach maturity and investors receive payments linked to outcomes, a further set of questions arises: how to compare results across SIB deals? What is the relationship between financial return and social impact? The promise of SIBs (and impact investing in general) is to align financial return with social impact, but one does not always imply the other.

Last year the New York Times (Success Metrics Questioned in School Program Funded by Goldman) reported how early-education experts questioned the validity of the impact results associated with a Utah pre-school programme designed to help 109 ‘atrisk’ kindergarten children avoid special education. Goldman Sachs reported a $260,000 payout from the SIB but, according to the NYT, critics argued that the programme’s unusually high success rate (99%) was based on a ‘faulty assumption that many of the children in the programme would have needed special education without the preschool, despite there being little evidence or previous research to indicate that this was the case’.This issue arises when validation of outcomes is not dependent on a control group or counterfactual assessment and there is the possibility of ‘creaming’, that is, enrolling participants who are easiest to work with and to achieve outcomes. As not all SIBs will run a control group – it is not always practical, ethical or cost-effective – impact investors must appreciate the limitations of linking the achievement of an outcome (say avoiding special education) to impact (the degree to which the outcome resulted from the intervention). In the case of the ThinkForward SIB, while the payments were not linked to a counterfactual, we worked with the Education Endowment Foundation to carry out a pilot randomised controlled trial (RCT) with randomisation at school and pupil level. This evaluation, which is separate from the SIB work, aims to identify quantitative evidence that the ThinkForward intervention is having a positive impact on education attainment and other outcomes. A clear relationship between financial return and impact is also complicated by some pricing strategies.

Sometimes commissioners link payments to a set of outcomes. In the case of the DWP Innovation Fund, for example, its rate card set a price for ten outcomes per participant. DWP would pay for improved attendance at school (up to £1,400 per participant), entry into employment (up to £3,500 per participant) and for other outcomes linked to improved employability. The outcomes in the rate card are important, but not all are equally relevant. A sophisticated impact investor will want to know if the financial return is driven by less important outcomes, say passing one GCSE at school (DWP pays up to £1,100 per participant), or is the result of achieving more socially meaningfully outcomes such as securing employment for 26 weeks (DWP pays up to £2,000 per participant). Another challenge with some payment structures is that they pay for the same outcomes, even if the programmes or the people the programmes target are very different. For example, a 12-week employability programme may be effective with young people who are work ready, but those further from the labour market may require longer interventions to achieve the same employability outcomes. The pricing structures don’t always recognise these nuances. Over time we hope that commissioners’ pricing will be better targeted, reflecting different levels of support needed to get different groups of people to the same outcome. Until then, investors who want to support the hardest to reach may need to accept lower financial returns. Social impact bonds have opened up government funding streams to delivery partners that would otherwise not have been able to access payment by outcomes financing, and nudged up expectations of managing to impact. However, the challenges reviewed above suggest that SIBs should not be seen as the perfect tool for measuring social impact. Ultimately, investors, commissioners and delivery organisations must define and measure success, and hold themselves accountable to the objectives laid in their own organisational missions.

Our three golden rules for reporting

Street League’s fantastic new campaign calls for more charities to address low public confidence in the sector by being more transparent around impact.

To truly support those in need we too have to accept that sometimes we don’t always get the results we want and we have a duty to tell this with accuracy and transparency. Balancing the powerful stories of our young people alongside the bare facts is a key part of this. Which is why like Street League we also pledge to uphold to the three golden rules for reporting:

1. We will never over claim what we do:

We all need to be very aware of this in how we use statistics. There is increasing pressure from funders to be seen to be bigger and working with larger numbers.

We specifically target young people most at risk of future unemployment, who need the most support in order to make a successful transition from school to sustained employment. Nearly three years ago we made a decision to halve the number of young people our Progression Coaches work with in order to allow them to provide the support necessary to support this high risk group.

Instead of making claims of working with large numbers, we instead focus on the ‘distance travelled’ and intensity of support needed to achieve this.

We work with external, independent evaluators to undertake studies of the impact of ThinkForward. We have recently received the result of two evaluations, and will be sharing some of the findings and learnings from these in blogs over the next few days.

2.      All percentages are backed up by absolute numbers to avoid being misleading:

We have a huge range of data on the more than 1400 young people we have supported over the past five years. This includes information on their improved attendance and behaviour at school, results in qualifications, progression to college or employment, and whether this was sustained. We report on this to funders and other stakeholders, and look to provide both absolute numbers and context (eg survey response rates) when doing so.

3.      All our outcomes could be independently audited:

We ask all our Coaches to collect evidence from third parties regarding outcomes. This might be letters from employers confirming jobs, copies of pay slips or exam results, or school data on changes in attendance or behaviour.

Previously, a large proportion of our funding came through a social impact bond, which required us to submit the above evidence to receive funding. This ended in 2015, but we still feel it is vital to retain this external evidence of the impact of our work.

Using external organisations to undertake evaluations of the impact of ThinkForward helps ensure these studies are independent.

 

ThinkForward reports on the impact of our activities in a range of ways which are consistent with Street League’s Call for Clarity and with the broader responsibility which we, like Street  League, believe that the charity sector has to be clear and transparent when reporting our impact.

– Article written by Luke McCarthy Programme, Development Manager at ThinkForward