Introductory video - Description of the section
Impact Assessment Methodology
This document summarizes the AfriConEU impact assessment methodology. It offers an overview of the purposes of impact assessment within the AfriConEU context, along with a short description of the model that is employed for assessing the Academy’s impact and a timeline for the application of the impact assessment plan. The methodological framework was developed in close collaboration with the AfriConEU partners, who were in charge of designing the AfriConEU Academy Capacity-building and Transcontinental Partnerships programmes and it is implemented with the contributions of all partners involved in the Academy implementation phase.
Scope
The purpose of the AfriConEU impact assessment methodology is to evaluate the performance of the AfriConEU Academy against agreed objectives and to record the Academy’s impact on Digital innovation ecosystems in Africa. Therefore, the focus of the methodological framework presented in the current document is limited to measuring the effects of the AfriConEU Academy’s flagship programmes, rather than the success of the project as a whole. Thus, the evaluation of some of the project’s tasks and activities goes beyond the scope of the current document.
Model
The model that was adopted for the development of the AfriConEU impact assessment plan is the Kirkpatrick model for training evaluation. One of the main reasons for this decision was the fact that the Kirkpatrick approach allows much space for in-depth analyses of the Academy’s impact, as it examines its effects on four distinct levels covering short-term, mid-term and long-term performance and behavioral shifts on both individual and organizational level.
Another important factor leading to the selection of the Kirkpatrick evaluation model, is its adaptability to different context as, although it is tailored for evaluating training programmes, it can easily be modified to measure the impact of the Academy’s networking activities, while maintaining its multi-level impact assessment perspective.
Level 1
The first level of evaluation focuses on participants’ experience, by measuring their satisfaction with the event, their engagement during the activities, as well as their perceptions of the event’s relevance for their work. The purpose of these measurement is to identify whether the Academy’s events are structured and implemented in a way that motivates attendees to actively participate in the activities and get back for more AfriConEU events.
Level 2
On the second level of evaluation, the Kirkpatrick model examines learning effects of the training, in terms of raising participants’ knowledge and skills as well as in the participants’ attitudes towards the training content. The idea behind Kirkpatrick’s approach is that no training programme can have any actual impact, unless trainees implement what they learn on their everyday work practices. Therefore, measuring short-term effects on trainee’s attitudes, confidence and commitment to implement their acquired skills and knowledge on-the-job is equally important to the activities’ actual learning effects.
Taking into consideration that the AfriConEU Academy is not limited to capacity- building activities with an inherent learning orientation, but also features a Transcontinental Partnership programme with different objectives, the AfriConEU impact assessment methodology is adapted to the nature and purposes of each activity. Thus, rather than measuring learning effects, the impact assessment plan explores how the Transcontinental Partnership activities contribute in helping participants connect with like-minded people, expand their networks and establish professional relationships that can lead to future partnerships.
Level 3
The main focus of the third level of evaluation in Kirkpatrick’s model is on identifying whether involvement with the programme’s activities brings about any actual behavioral shifts on individual and organizational levels. More specifically, Level 3 aims at identifying behaviors that are considered critical for the organization’s success, and to investigate whether what are the mid-term and long-term effects of Academy’s activities on the anticipated changes in everyday work practices.
Level 4
The final level of Kirkpatrick’s model explores the impact of the Academy’s activities on participants’ organizations actual results, by examining whether they have achieved a series of expected outcomes. Level 4 evaluation is based on a thorough analysis of the Academy’s objectives, through which they were translated into measurable targets.
Impact Assessment Timeline & Process
Phase 1: Comprehensive analysis of each programme’s/ subprogramme’s objectives, and their translation into measurable targets. Development of Expected outcomes checklist.
Phase 2: Development of generic questionnaires to assess short-term impacts of every event (Level 1). Analysis of specific objectives of all AfriConEU activities and development of event-specific evaluation tools (Level 2). Initiation of Data collection process in individual Academy activities.
Phase 3: Analysis of event-specific evaluation findings to identify relevant themes for further exploration and critical behaviors for organizational success and develop interview guides. Initiation of data collection process for expected outcomes.
Phase 4: Follow-up study with qualitative interviews with representatives of involved stakeholders.
Phase 5: Overall analysis and development of final report.
