A Digital Data Ecosystem to evaluate blended CPD in Rwanda | by VVOB

Together with the University of Rwanda – College of Education (UR-CE), Rwanda Basic Education Board (REB), VVOB – education for development offers CPD programmes to educators in Rwanda. 

As part of these programmes, VVOB builds the capacity of REB and UR-CE to evaluate the CPD programmes. The evaluation system relied on standalone, paper-based instruments which limited the ability to gain access to real time data and to connect datasets. To improve the evaluation system, VVOB and partners developed a digital data ecosystem. 

The system is made up of interconnected information technology resources that allow for a fully digital process, from data collection to reporting. The digital data ecosystem has resulted in fewer errors and faster feedback loops. The system is built on a framework developed by Kirkpatrick & Kirkpatrick (2015) which states that an evaluation of a training programme (e.g. CPD programmes) should focus on four levels:

  1. Level 1 Reaction: this level focuses on the degree to which participants react favourably to the CPD. Examples of indicators are satisfaction, engagement and relevance. 
  2. Level 2 Learning: this level focuses on understanding whether the CPD has increased the knowledge and competences of participants. In addition, it is also important to evaluate commitment and confidence at this level, as these two indicators help to close the gap between learning and behaviour. 
  3. Level 3 Behaviour: this level focuses on the degree to which participants apply what they have learnt on the job. 
  4. Level 4 Results: this level looks at whether the CPD has had an impact on the overall (school) environment.Examples of indicators are improvements in the application of the five standards of school leadership and an improvement in school culture. 

In addition to these four levels, an extra level has been added: Level 0 Participation. This level studies the level of participation in the blended CPD programme. 

The digital data ecosystem is made up of components that are integrated in a dashboard that includes Key Performance Indicators for each level. The process to generate the dashboard has been digitised. Figure 19 shows the digital data ecosystem architecture, the software that has been used and how it is being applied.

How Right to Play evaluates the outcomes of the Partners in Play (P3) Project

The goal of the Partners in Play (P3) project is to improve the quality of education for Ghanaian girls and boys aged 4-12 through a scalable and replicable Learning through Play (LtP) model. 

To achieve this, improving the capacity of the Ghanaian education sector to integrate LtP to deliver the curriculum is seen as an important step. One capacity building activity in this programme is the Reading Through Play e-training course, in which the first-line beneficiaries are primary- and secondary-grade teachers.

For the monitoring, evaluation and learning component of this course, the quality of the course content and the delivery of the e-training are evaluated. This is done by measuring improvements in knowledge, attitudes and practices of participants.

The LMS platform which hosts the e-training course has an embedded reporting system that tracks reach and individual progression. In the areas where the LMS cannot evaluate a user’s behaviour and attitude, a pre- and post-survey is directly incorporated in the e-training course that must be filled out by each participant.