Description of the programme
Evaluation and monitoring are essential elements of assessing the effectiveness of public policies, benchmarking against desired outcomes, and helping to enact evidence-based actions to improve these policies. National strategies for financial literacy are complex, multi-year, multi-stakeholder public policy projects that can strongly benefit from comprehensive evaluation designs.
The report provides case studies from across the world and outlines a model template for structuring the process of designing an evaluation of a strategy by using a mapping of outcomes and a theory of change. These elements are aimed at providing guidance to policy makers interested in evaluating their national strategies for financial literacy.
The study
The 2020 OECD Recommendation on Financial Literacy recommends that countries establish and implement national strategies through a policy approach that incorporates monitoring and evaluation to assess the progress of the strategy, and to propose improvements.
This report summarises and draws lessons from the evaluation approaches and activities of 29 countries and economies that are part of the OECD International Network for Financial Education. It provides implementation guidance on the evaluation of national strategies for financial literacy. It also discusses reasons for evaluating national strategies, good practice approaches and methodologies, associated challenges and benefits, as well as insights on funding the cost of evaluation and on post-evaluation communication plans.
This report summarises the results from multiple sources of data: a questionnaire on evaluations of national strategies; information obtained through interviews with OECD/INFE Members; the peer review of the UK’s “What Works Fund”, and information gathered through meetings of the OECD/INFE Working Group and Technical Committee. It also builds upon the broader OECD literature on policy evaluation. The primary research was undertaken in 2019 and 2020.
Key findings
- The most common evaluation approach included a mix of quantitative and qualitative approaches.
- The quantitative approach was found to commonly involve conducting a financial literacy survey at regular intervals, as well as monitoring a broader set of quantitative indicators (such as access to financial services, participation in financial literacy and education activities, and change in financial knowledge of participants). More advanced quantitative approaches combined a number of indicators into a tracker, exploring sentiments and behaviours.
- Qualitative approaches tended to revolve around gathering stakeholder feedback and engaging stakeholders into discussions or focus groups.
- Around 60% of national strategies had overall goals linked to a broad set of quantitative indicators (such as access to financial services, participation in financial literacy and education activities and change in financial knowledge).
- Around a quarter of national strategies had aspirational goals (such as improving the relationship of their citizens to finance and helping them achieve their financial goals) that were not directly linked to measurable quantitative objectives.
- The most common objectives were financial education and literacy ones; present in all the national strategies of respondents. About half of the national strategies had financial well-being (50%) and financial inclusion (45%) related objectives.
- Funding for evaluation activities comes overwhelmingly from the resources of the institutions in charge of financial literacy in the respective jurisdiction. Few jurisdictions (less than 30% of the respondents) have dedicated budgets for evaluation activities.
- Required reporting on evaluation activities (to a higher institutional authority) that looked at progress towards the objectives was done in almost half of the participating jurisdictions. Reporting on the contributions of the national strategy towards broader economic or financial objectives was less common.
- The report proposes a ‘blueprint’ to inform the development of an evaluation process. The key pillars are:
- Mapping the financial landscape, determining the evaluation needs and priorities.
- Designing an evaluation plan for the national strategy with clearly assigned lines of responsibilities, and transparent and multiple flows of data.
- Creating an engaging and inclusive evaluation process that fosters an evaluation ecosystem, with clear incentives for accountability and communicating widely the evaluation results with the aim of popularising success.
Recommendations for policy makers:
- Taking a system-wide approach, incorporating stakeholder implementation
- Creating a culture of evaluation in the jurisdiction
- Thinking about the evaluation early, while preparing the national strategy
- Having an explicit theory of change which engages stakeholders
- Using mixed evaluation methods
- Flexibility to change in response to changing needs and context
- Having a comprehensive and engaging communications strategy
- Engaging professional and external evaluators.
Points to consider
-
Methodological strengths/weaknesses: The report summarises key quantitative data from 29 countries, helpfully contextualized with references to the evaluative practices of specific countries. The full report therefore contains considerable detail which may be helpful to refer to in full.
-
Generalisability/ transferability: This meta-evaluation covers a range (but not all) of OECD countries, which are in themselves often regarded as high-income economies. Care should therefore be taken not to assume they are representative of all nations.
-
Relevance: The report summarises key quantitative data from 29 countries, helpfully contextualized with references to the evaluative practices of specific countries. The full report therefore contains considerable detail which may be helpful to refer to in full.
