ThinkForward took part in the randomised control trial (RCT) to help the EEF test the effectiveness of two different RCT methodologies which would then inform the design of a larger randomised control trial. The RCT was conducted over a two-year period across two academic years.
Randomisation took place at both school and pupil level. Regarding the methodology, the trial found that the most effective randomisation should be at a school level. On the educational attainment of young people, the study did not find evidence of improvement in GCSE attainment. However, teachers, Coaches and young people from across both schools reported that they believed the programme was beneficial and the study notes improvements in young people’s behaviour.
1. Why did ThinkForward participate in the Education Endowment Foundation’s (EEF) randomised control trial (RCT)?
ThinkForward took part in the RCT to help the EEF test the effectiveness of two different RCT methodologies which would then inform the design of a larger randomised control trial. ThinkForward has always been focused on ensuring their programme design has a long term and sustained impact on young people’s transitions from school to work. At the end of our first two years of delivery, the EEF’s trial provided us with a unique opportunity to take part in a cutting edge evaluation, to learn from our initial programme design and to make improvements ahead of further evaluation. Shortly after the start of the RCT in our third year we went through a rigorous Theory of Change process which led to refining our mission to focus on young people most at risk of unemployment and reducing each Coach’s caseload by half (from 100 to 50).
2. How was the RCT designed?
The RCT was conducted over a two year period across two academic years. Randomisation took place at both school and pupil level. Within the two randomly assigned schools, eligible pupils were randomly allocated either to a ThinkForward group or a within-school control group. Across the four schools, there were 181 pupils in the year 11 group, 40 of whom received ThinkForward support, and 160 pupils in the year 10 group, 37 of whom received ThinkForward. The study focused on the first two years of ThinkForward’s five year programme and young people’s educational outcomes.
3. What were the key findings?
Regarding the methodology, the trial found that the most effective randomisation should be at a school level. Due to the ‘spill-over effects’ on other pupils in the school it does not work to have the control group from the same school.
On the educational attainment of young people, the study did not find evidence of improvement in GCSE attainment. Year 11 pupils received 6 months of support from their Coach before their GCSE examinations, in preparing for their GCSEs young people’s primary focus is on their revision and exam preparation and it is often harder for Coaches to work one-to-one. According to ThinkForward’s refreshed programme design, participants would have received 2.5 years of support before taking their GCSEs.
Teachers, Coaches and young people from across both schools reported that they believed the programme was beneficial and the study notes improvements in young people’s behaviour.
More work needs to be done on the effective collection of data when conducting a trial of this kind and at greater scale. ThinkForward welcomes the recommendation that in a future trial, data collection is undertaken by a third party rather than Progression Coaches.
4. What has ThinkForward learned from the findings?
ThinkForward’s approach is to be open to the lessons we can learn from different types of evaluations and their findings. We are focused on making changes in two key areas covering programme design and delivery and future evaluation.
In terms of our programme design, following our Theory of Change process two years ago, we now enroll young people at the start of year 9 so that there is more time (2.5 years) to work with them ahead of their GSCEs, increasing the likelihood of impact on attainment at 16. We have also halved the caseloads of coaches, to increase the intensity of the support that young people receive. To strengthen our delivery we have captured our refined programme design in practical materials to support delivery, including our Coach Handbook.
We are also focused on improving our partnerships with schools, all schools have Service Level Agreements in place to ensure effective data sharing and access to dedicated and confidential space for Coaches to work with young people.
In future we would like to conduct evaluations using a much larger sample size. This was a small study, only 77 young people received the programme vs. 264 in the control group which meant that the required level of impact to show significance was very high.
5. What happened to the young people who participated in the trial?
Half the young people who participated in the trial graduated from ThinkForward in August 2016 with impressive results. The other half will graduate from ThinkForward in the summer of 2017. Across the two groups 92.4% are in positive destinations including education, employment or training. 6 young people progressed to university, 17 into employment and the remainder are continuing their studies.
Five young people (7.6%) who took part in the trial are currently not in education, employment or training (NEET). This is below the NEET rate for the geographical area in which the trial schools are located. This is an impressive outcome given that the young people were selected as being at high risk of unemployment.
ThinkForward continues to give targeted support either through targeted support by WorkForward, ThinkForward’s employment brokerage and one-to-one support from their Progression Coach.