Blog...

Understanding the SIB’s success by looking at the (right) numbers

As we analyse the results from the social impact bond, it has made us think afresh about our understanding of success in relation to social investment.

Much coverage of the first SIBs has focused on their financial achievements.  In many ways this isn’t surprising, as it is the ability for small charities to deal with payment-by results contracts, by taking on an investment and then paying it back with a return, which makes them usual.

In the case of ThinkForward, we are proud that we have been able to pay back our investors, Impetus-PEF and Big Society Capital, not just the initial capital they invested in us, but also a small return.  As social investors they will recycle that money and it will be used to help other similar social initiatives in the future.  However, making a positive financial return, as good as it is, doesn’t necessarily mean that a long-term social impact has been achieved.

In part, measurement of social impact is down to the outcomes specified by commissioners.  In the case of ThinkForward’s SIB, that was the Department for Work and Pensions (DWP), who have a menu of payable outcomes, based on known predictors of future employability.  These range from improving young people’s attitude to school, through to helping a young person sustain employment for six months.

On these measures, ThinkForward does well.  While Statistics Authority rules mean we can’t share any data on the SIB before the DWP’s official release next year, ThinkForward has more generally in 2014/15 helped:

•    85% of 14-16 year olds to improve their behaviour or attendance at school;
•    60% of 16 year olds to achieved good GCSEs – twice as many as their schools initially predicted; and
•    96% of 16-18 year olds to progress into further education, employment or training.

There is no doubt that our staff worked hard to support young people and that these results are fantastic for the young people concerned.  Nevertheless, we often ask ourselves questions about how socially significant they really are.  Do these outcomes have any correlation to their likely future employability?  Would they have happened anyway?

The first of these questions is easier to answer.  Not participating in education, employment or training is associated with negative outcomes later on in life, such as unemployment, low pay and depression, while participating and gaining qualifications has a positive impact in terms of employment and income.  DWP recognises that many of these long-term outcomes could take several years for the benefit and wider fiscal and social savings to accrue. They therefore pick intermediate outcomes that are most clearly linked in evidence-based research to effective longer-term employment outcomes.

The second question is harder.  Every programme inevitably includes some ‘deadweight’, young people who might have got into work anyway.  The only way of measuring this is to establish a counterfactual using an approach such as a Randomised Control Trial.  We are fortunate that the Education Endowment Foundation has been supporting us to conduct one and we look forward to seeing the results next year.  In lieu of these, the onus is on all organisations like ThinkForward to ensure it working with those displaying the greatest ‘at risk’ indicators and who need our support the most.  In our case, 50% of the young people we support have lower than average academic attainment and are not predicted to get the five GCSEs A*-C that makes them seven times more likely to get into employment.

Perhaps most of all, charities need to measure their impact in the context of their mission – what they say they will do, and for whom.  For ThinkForward that means we will keep tracking young people’s progress for at least 18 months after they leave our programme; far longer than DWP require, but for us a proper measure of whether their employment is sustained.  In this, the social impact bond was a means to an end, not the end itself.