evaluation

Changing behaviour around online transactions

Evidence type: Evaluation i

Description of the programme
The study
Key findings
Points to consider
Full report

Description of the programme

Digital exclusion compounds the poverty premium. Individuals who do not make use of online retail and digital financial services are more subject to inflated prices, are less able to compare prices, and are unable to take advantage of online-only savings.

Community financial inclusion interventions are impactful but rarely focus resource on digital inclusion. Conversely, the fintech sector creates digital tools for financial inclusion without supporting the skills that people need to use those tools. This project sought to create evidence around how to embed digital skills in a financial capability intervention.

Good Things Foundation and Toynbee Hall ran this project which was delivered by member organisations of the Online Centres Network. It ran between November 2016 and December 2017 and was delivered to struggling working-age adults in mainly deprived wards across the country. The project tested the efficacy of an assisted digital transaction, by supporting project participants – who used their own money - to conduct a live online transaction.

The study

The aim of the evaluation was to create evidence that an assisted digital transaction could change transactive behaviours. It did this by exploring the impact of the intervention in relation to four key outcome areas:

Ability to transact online independently
Use of online financial products and information
Ability to exercise judgement in financial decisions
Ability to maximise income

The researchers evaluated the project by structuring it as a Randomised Control Trial (RCT), using a cluster stepped-wedge design. Participants completed pre and post-intervention surveys, which measured self-assessment of financial capability, consumer behaviour, budgeting, self-assessment of financial wellbeing, financial outlook and digital inclusion. Of the 645 participants registered on the course, the researchers collected complete outcome data for 630 participants.

Alongside the surveys, the researchers also conducted qualitative interviews with participants in ten of the centres, between May and September 2017.

Key findings

Assisted digital transactions improve self-rating of financial capability:

Participation in the assisted digital transaction improved scores across six key indicators. The combined score across these indicators increased by 9% from baseline through to post-intervention. This is a statistically significant effect.

Assisted digital transactions change transactional behaviour:

People who were part of the intervention were 6.5 times more likely to transact again if they were supported to transact through the intervention. Again, this was a statistically significantly effect.

Points to consider

Methodological considerations:

There was drop-off in recruitment towards the end of the project, which researchers felt was likely down to exterior factors (such as Ramadan, Eid and the school holidays). A crossover cluster RCT design, rather than a stepped-wedge design, may have better accounted for drop-off towards the end of the project.
The project ran to strict time limits, meaning that they had little flexibility to extend the delivery period to mitigate non-recruitment or poor attendance.
It is not clear how many people were interviewed for the qualitative research, and how they were sampled.

Transferability:

The project delivery model provided open-ended support to participants. To make the assisted digital transaction transferable, other agencies need to be aware of the additional support needs that the transaction creates.