What is Predictiv?
Predictiv is an online platform for running behavioural experiments built by the Behavioural Insights Team. It enables governments and other organisations to run randomised controlled trials (RCTs) with an online population of participants, and to experiment whether new policies and interventions work before they are deployed in the real world. To date, we have conducted more than 75 trials in 12 countries.
How Predictiv works
Methodology
Where appropriate, we use online randomised controlled trials (RCTs), which are widely considered to be the gold standard of impact evaluations. In an RCT, people are randomly assigned to two (or more) groups. One group receives the policy or material being tested while the other (the comparison or control group) receives an alternative treatment, a dummy treatment (placebo) or no treatment at all. The random assignment to different groups means you can compare the effectiveness of a new intervention against what would have happened if you had changed nothing at all (the counterfactual).
Predictiv does the same thing, by randomly allocating people from an online panel to different versions of communications materials, product, or service, such as: differently worded letters; different forms of terms and conditions for a new service; different images or videos for a communications campaign; or different versions of a website or app. We then measure participants’ comprehension, ability to choose the best option, and other types of outcomes, and compare these versions against each other using statistical analysis techniques. Predictiv can provide robust evidence much faster than alternative solutions such as field RCTs: our data collection is normally completed within 1-2 weeks.
Example: participant journey through an online RCT on Predictiv
Technical capabilities
The Predictiv platform is highly flexible and can deliver a variety of interventions including text, graphics, video and audio. Predictiv has the technical flexibility to replicate a wide range of decision environments ranging from simple letters to complex website simulations.
Another unique feature of the platform is variable incentives: we use best practice for online experiments to tie participant incentives to correct answers (where possible) to elicit more reliable responses.
Example: simulation of online shopping checkout page
Experiment participants
Once customers have identified what they want to test, we can set-up the experiment and recruit participants from our large international panel spanning more than 60 countries. The total participant pool in the UK covers more than 500,000 people who are roughly representative of the general UK population based on gender, age, location, and income. Beyond the UK, Predictiv has access to over 1,000,000 adults in the rest of Europe, 2,000,000 in the US, and 200,000 in China. Basic demographics, such as gender, age, household income, and location are available for all participants. Additional screening questions, such as whether someone currently uses a particular product or service can be created if necessary.
We use best practice for online experiments to incentivise participants to elicit more reliable responses. Variable rewards are a cornerstone of experimental economics because they ensure that choices in the experiment have consequences. They can reduce the likelihood of participants giving socially desirable answers, focus attention, and can increase effort, particularly compared to no-stake environments.
Experience
Our clients include government departments and agencies, as well as large financial services companies. Questions that Predictiv has answered have included the following:
- Would increased transparency about foreign exchange transaction fees help consumers make better choices and save money?
- Which letter best explains the implications of a government programme involving data sharing, while minimising customer opt-outs?
- Is a savings prompt by retailers effective at increasing savings for people who currently have no or little money saved?
Running tests on Predictiv
Running an experiment on Predictiv is rapid. From the moment that the research question has been designed and the interventions have been developed, a Predictiv experiment can take as little as 10 weeks to run. Each Predictiv test includes:
- Advice on experimental design,
- Technical setup of the test,
- Participant recruitment and incentives,
- Data collection and analysis,
- Results report & recommendations.
Example project timeline, from the moment we define the research question.
Predictiv tests
1. Comprehension: How well does my target population understand a product or service?
Aim: The comprehension test looks at how well people understand a piece of information, such as a letter about making a tax payment, a communications campaign or features of a new product or service. A comprehension test can also focus on applied problem solving, asking people to use the information they’ve been provided to answer a practical question such as how much the price of goods is likely to increase in a year given a 2% inflation rate.
What do I learn from this test?
- Which version of communication material helps your target population to best understand your key message.
- Which parts people struggle the most to understand, such as knowing who the message is from or what they have to do.
- Whether people understand the main point even when they are not paying much attention.
Case Study: Increasing understanding of the data sharing agreement for a return-to-work programme in Greater Manchester
Working Well is a programme in Greater Manchester to support people getting back into work. It offers tailored support, but the services require clients to share personal data. Currently, people are asked to go through a 5-page Privacy Notice before signing up.
We tested two new versions of the Notice to see if we could retain (or even increase) comprehension while substantially reducing the amount of information. The first new version simplified the original messages and cut down the text to 2 pages. The second new version used images to explain the processes visually (with accompanying explanatory text). Both of these new versions significantly outperformed the original version.
2. Choice set: How does my target population evaluate different product alternatives and do they make the right choices?
Aim: The ‘choice set’ test presents people with a number of different product options and asks them to pick the one they think is the best.
What do I learn from this test?
- Whether a new way of presenting product information improves decision-making quality, such as choosing the healthiest breakfast cereal under different nutrition labels.
- How likely people are to stick to a default option, and what can increase switching.
- Whether brand information impacts decisions and what information people access in making their decision.
- Which messengers are the most effective at changing choices.
Case Study: Increasing transparency about fees in foreign exchange transactions
Predictiv supported a client in building evidence on consumer transparency in the FX market. We looked at what pieces of information could help consumers to choose the best FX deal defined by total £ in their pocket following the transaction. We used a choice set to test 3 alternatives to a ‘low information’ industry practice of displaying the fee structure in FX transactions.
3. Bespoke Tests
Predictiv also has the possibility to accommodate more bespoke tests. If you’d like to hear more please get in touch with Abi Mottershaw abigail.mottershaw@bi.team.
