If you’ve ever run a Randomised Controlled Trial (RCT), you know how exciting it is to find a solution that works. However, as solutions are implemented and scaled, they are often applied to different contexts than the one they were originally developed for and tested in. This can impact results. For example, RCTs looking at the impact of body-worn cameras on the use of force by police officers found that effects varied significantly between trials: some found no difference in the use of force between officers wearing cameras compared to those not wearing them; in other trials, officers with cameras were actually more likely to use force.
How can we be sure that our solution continues to deliver positive impact as we scale?
At BIT, we believe that good evidence is essential to scaling good solutions. Our Predictiv platform makes it easier for policymakers to test solutions in different contexts and for different target groups. Recently, we tested and extended successful solutions to improve online terms and conditions (i.e., the bits we always say yes to, but never actually read) from the UK to the Netherlands.
In both countries, we found that our interventions increased understanding of key messages and engagement with T&Cs. The patterns across both countries are similar, though there are some differences in the size of the effect. By running the replication, we are more confident that our results will hold for Dutch consumers and it also helped us identify new solutions that we can test in future trials.
Predictiv and our T&Cs experiment are available for other organisations to use. If you want to hear more, please get in touch with us!
The details: Our solution to improve online retail T&Cs
It has been estimated that it would take 76 work days to read all the privacy policies we encounter in one year. The lack of understanding and engagement with T&Cs can contribute to consumer harm. In one study, 100% of the sample agreed to contractual policies that included immediate payment via the User’s first-born child.
In 2018, we partnered with the Department for Business, Energy & Industrial Strategy (BEIS) in the UK to run six online trials using Predictiv to test different ways of presenting T&Cs with the aim of encouraging more consumers to read them and understand them. You can read all about our results here in the Best Practice Guide.
In a separate project, we worked with the Netherlands’ Authority for Consumers and Markets (ACM) to replicate successful interventions in the Dutch retail environment. We used three approaches:
1) Replication across geography: We used the same intervention, key terms presented with visual icons, in the Netherlands that we used the UK. We tested it in the same setting – making an online purchase.
2) Replication across task: We took one intervention, an urgency notification to alert the consumer, designed to increase engagement with a privacy policy when setting up an online account in the UK, and applied it to increase engagement with terms and conditions when making an online purchase in the Netherlands.
3) Extending the original trial: We combined one intervention (icons summary) that increased comprehension of terms and conditions with a second intervention (urgency cue) that increased engagement with a privacy policy to understand if we could increase both comprehension and engagement simultaneously.
