Proving our ‘Theory of Change’
The formal evaluation of programmes is important if we want to understand what works and what could be improved. During the lifetime of Inspiring Purpose, there have been several informal evaluations and a significant amount of testimony collected from both teachers and pupils. These have indicated a transformational effect on those who take part.
In 2016/17, leading social scientists from Montclair State University USA, working in partnership with IPSOS MORI will use ground-breaking methodology to study the programme effect in detail. This new research will investigate how our young people set goals, demonstrate future-mindedness and develop a sense of purpose. It will involve 40 S2 classes from 30 schools throughout Scotland. This new research was preceded by a pilot study undertaken by Montclair State University during 2014 and 2015 which involved the Character Scotland team along with staff and pupils from Kemnay Academy, Gracemount High School, Inverurie Academy, Turriff Academy and Clydebank High School. The study is described below and the full report is available on request from the Character Scotland team.
THE RESEARCH STUDY
Inspiring Purpose was evaluated in two parts. First, REDSS Lab at Montclair State University collaborated with Character Scotland to determine the program’s theory of change, the hypothesized processes through which the program affects young people. The collaboration yielded a pathway model that visually represents these processes of change. Second, REDSS Lab conducted a process evaluation, to assess various components of the program. The diagram below show's our 'Theory of Change' for Inspiring Purpose. The yellow and pink pathways represent the core storyline of Inspiring Purpose.
METHODS AND APPROACH
A variety of data sources were used to evaluate Inspiring Purpose including teacher interviews, pupil interviews and pupil surveys, as well as posters completed by program participants. Pupil interviews and surveys were completed before and after pupils participated in Inspiring Purpose. Survey measures examined positive youth development (PYD), goal setting, future aspirations, future orientation, and sense of purpose. Poster quality was assessed by a systematic scoring tool for posters based on degree of completion, development, and alignment of poster responses. Teacher implementation approaches were compared to pupil poster quality and outcomes.
Nearly all pupils (94%) who participated in Inspiring Purpose generally completed the entire poster. A strong association was observed between the values selected and pupil’s future aspirations (alignment of these poster sections ranged from 41% - 84%). In addition, there was also a strong connection between pupils’ visions for a better world and their plans to personally bring that vision to life (these sections were well matched in 76% of posters).
Pupil survey data revealed very little change in outcome scores between Time 1 and Time 2. Reasons for stability across the pre- and post-survey measures could be attributed to the small sample size, measurement error, and/or the relatively high survey scores at Time 1 (thus, there was little opportunity for growth).
The findings from the qualitative analysis of teacher and pupil interviews helped enhance the quantitative results and offered insight about emerging trends. Teaching style and experience implementing Inspiring Purpose varied widely across the teachers interviewed. Teachers with more experience with the program tended to focus more on supplemental activities during their instruction. Students whose teachers spent more class time on the poster spent more hours overall working on their posters than students whose teachers did less classroom instruction related to the poster. Despite differences across teachers in implementation and delivery, students overall demonstrated understanding and development of values and virtues, such as connection and competence, although the actual word “values” was sometimes difficult for students to understand. In addition, goal setting, as well as the optimization of goals, were often articulated by pupils. Goal setting was strongly connected to various other concepts, such as future-mindedness and sense of purpose.
- The highest quality posters were produced by pupils whose teachers spent significant time on poster related activities in class but did not immediately have pupils work on the poster template.
- Preparing the students to work on the poster by using activities that lay the foundation for the poster elements is important.
- Posters are of higher quality when the poster is completed in school rather than at home.
- Teachers who fully integrated Inspiring Purpose into the broader curriculum tended to have pupils with higher-quality posters.
- Teachers suggested that Inspiring Purpose dovetailed nicely with their courses on religious and moral education.
- The resources offered to both teachers and students can be further enhanced to be more geared to the abilities of younger students, as well as to address the needs of teachers who are interested in additional implementation guidance.
ABOUT REDSS LAB AT MONTCLAIR STATE UNIVERSITY
The Research on Evaluation and Developmental Systems Science (REDSS) Lab explores questions related to developmental science and program evaluation and planning from a systems science perspective. Co-directors Dr. Jennifer Brown Urban and Dr. Miriam Linver, Associate Professors of Family and Child Studies at Montclair State University, engage in interdisciplinary projects and actively collaborate with researchers and practitioners across the US and in Scotland.