Appendix 2: Evaluation questions, indicators and data sources
Midterm Evaluation of the Pandemic Preparedness Strategic Research Initiative
| Issue and Question | Indicators | Data sources |
|---|---|---|
| 1. Effectiveness of priority setting | ||
| 1.1 To what extent were the PPSRI research priority setting processes appropriate, in terms of timeliness, mechanisms, and inclusivity of a) research areas and b) groups of researchers? | Stakeholders' views of appropriateness Researchers' perceptions of COI |
Key informant interviews: Section 2 Researchers survey Q4-9 |
| 1.2 How successful were the research priority setting processes – to what extent did they arrive at the most appropriate set of priorities for Canadian research in pandemic preparedness? | Stakeholders' views of appropriateness Researchers' ratings of priorities |
Key informant interviews: Section 2 Researchers survey: Section E |
| 1.3 To what extent was national and international duplication of effort avoided, and complementarity enabled? | Stakeholders' views of appropriateness Researchers' ratings of priorities |
Key informant interviews Q2.2, |
| 1.4 How necessary, appropriate (in terms of role and composition) and effective was the Task Group? | Stakeholders' views of appropriateness and effectiveness Researchers' perceptions of fairness, COI |
Key informant interviews Q2.3 |
| 2. Effectiveness of partnership development | ||
| 2.1 To what extent has the PPSRI been successful in building national and international partnerships? Which partnerships have been most and least successful? Are there partnerships that should be developed but have not been? | Stakeholders' views of effectiveness | Key informant interviews Q3.1-3.3 |
| 2.2 What have been the impacts of partnerships on: a) coordination and integration of national and international research programming; b) resource leveraging; c) research duplication and complementarity? | Stakeholders' views of impacts Documentation of partnerships, resources leveraged |
Key informant interviews Q3.1-3.3 Review of documentation |
| 2.3 To what extent have the partnership structure and reporting strategy been effective in supporting the ongoing initiatives of partners, including the PPSRI? | Stakeholders' views of effectiveness | Key informant interviews Q3.4 |
| 3. Appropriateness of program design | ||
| 3.1 To what extent is the suite of activities and funding programs offered through the PPSRI allowing the achievement of program objectives? Which components are most and least successful? | Stakeholders' views of appropriateness, Researchers' ratings of incentive features Uptake/quality of response to program components |
Key informant interviews Q4.1 Researcher survey Q3 Administrative databases |
| 3.2 Is the overall strategy of strategic competition effective in ensuring that the most promising research is funded? Would an alternative strategy including the enabling "front-runners" (recognized leaders in the relevant fields) to continue to pursue relevant PP research) be more effective? Is this the most effective strategy to ensure short and long term capacity development? | Stakeholders' views of effectiveness Researchers' views of effectiveness |
Key informant interviews Q4.1 Researchers survey Q21-23 |
| 3.3 How effective was the communication strategy used to launch the funding opportunities? | Stakeholders' views of effectiveness Researchers level of awareness and ratings of effectiveness |
Key informant interviews Q4.2 Researcher survey Q1, Q3, Q10 |
| 4. Achievement of desired funding opportunity outputs | ||
| 4.1 To what extent have the funding programs generated expected and/or desirable uptake from the relevant research communities? 4.2 To what extent do the sets of funded projects cover the intended field of program objectives? Which objectives streams are more and less well represented across funded projects? |
Stakeholders' views of uptake and coverage Per competition and per program stream: No. of applications, no. of grants, success rate (including successful-unfunded and funded), amounts awarded; disciplinary diversity of applications and grants; national and international collaborations involved in applications and grants; Comparisons on the above variables between successful and unsuccessful applicants |
Key informant interviews Q5.1-5.2 Administrative databases |
| 4.3 To what extent will the funded projects result building in research capacity in areas related to pandemic preparedness (e.g., training of students, redirection of research foci, new investigators on teams) | Stakeholders' view of capacity development opportunities No. of students and fellows involved in applications and grants, by level per program stream No. of investigators newly involved Evidence of change of research foci |
Key informant interviews Q5.3, 5.4 Researcher survey Q13,14, 19, 20 Administrative databases |
| 5. Success of PPSRI networking and KT activities | ||
| 5.1 To what extent to which the PPSRI has successfully facilitated communication and networking among researchers involved in pandemic preparedness research? Have all relevant teams and individuals been provided with networking opportunities, and what has been the uptake? | Stakeholders' views of success Researchers' ratings of networking Documentation of participation in networking activities |
Key informant interviews Q6.1 Review of program documentation |
| 5.2 To what extent are communication and networking producing the expected results in terms of enhanced collaboration and increased capacity? | Stakeholders' views of success Researchers' ratings of networking impact |
Key informant interviews 6.2 Researcher survey Q11,12,15 |
| 5.3 To what extent has the groundwork been put in place for effective knowledge translation to occur (e.g., inclusion of end-users, KT plans)? | Stakeholders' views of success Proportion of funded projects with KT plans Degree and type of involvement of research users |
Key informant interviews Q5.4 Researcher survey Q17-18 Administrative databases |
[ next section ]
Supplemental content (right column)
- Modified: