The world in simulation – building evidence using an online laboratory

As a reader of our blog, chances are that you’ve heard of Daniel Kahneman. An Israeli economist and author, Kahneman was awarded the Nobel prize in 2002 for his seminal work, Thinking Fast and Slow, which explored human decision making from the prism of economics. Who you may be less familiar with, however is that of Vernon Smith. Smith actually shared the Nobel prize with Kahneman in 2002 for his use of laboratory experiments in economics and played a vital role lab experimentation in the field, a key factor in the rise of behavioural economics. However, policymakers and practitioners have often overlooked the versatile tool of online laboratories.

Thanks to online labs and the widening scope for experimentation, this is now changing. At BIT, online experiments have already supported over 65 projects, helping evaluate policy initiatives in areas such as gender equality, international development, and consumer protection. The UK’s financial regulator (FCA) also runs online experiments and has previously investigated questions such as insurance add-on sales and algorithmic investment advice.

In this blog, we give you a quick tour of the what and why of online (laboratory) experiments.

What is an online experiment?

An online experiment is a form of Randomised Controlled Trial (RCT), meaning that people are randomly assigned to treatment and control groups. The term ‘RCT’ is often used to describe field experiments, where real people make real decisions in the real world. Online experiments are RCTs that simulate real decisions with recruited participants. They focus on measuring choices that are predictive of real-world behaviour or the mechanisms underlying behaviour (such as trust or comprehension). Below is an example of an online experiment simulating a price-comparison website for pension products:

Why run online experiments?

• Greater flexibility: Experiments are powerful because they allow us to measure the causal impact of a new approach. Unfortunately, running them in the field can be difficult logistically and certain interventions may not be implementable for ethical or practical reasons. The online lab gets around these constraints because it gives researchers full ownership over the design of the choice environment and the materials that people see.
• Speed: With the right infrastructure, lab experiments are straightforward to set up and run, especially when they are conducted with online participants. This means that data can be collected within days or weeks, which allows for more ideas to be tested and can support evidence-based decision making when timelines are tight.
• More data: A big advantage of the lab is that it allows more (and sometimes better) data to be collected compared to other methods. For example, when BIT studied credit card repayment decisions in the lab, we measured not only what people chose to repay, but also tested how well they understood what level of repayment minimised their interest payments. It turned out that many people wrongly believed that the default ‘minimum payment’ amount was the best payment to minimise their cost of borrowing. Such insights into people’s comprehension and beliefs are crucial in understanding why they make certain choices.

What does the future hold for the lab?

Despite a rich history in the social sciences, online experiments have only just begun to support policymakers and practitioners in solution design and evaluation. With more aspects of people’s lives moving online, there are many reasons to be optimistic about the future of the online lab. Here are two trends we see as particularly valuable:

• Better targeting: More and more of our decisions happen in an online space, and individuals are increasingly likely to have continued access to the web via mobile devices. This creates more opportunities for online experiments to reach larger, as well as more targeted audiences, which large online panel networks help cater for. In addition, it creates opportunities for organisations to recruit their own customers into an online lab. For example, some fintech companies pre-test novel changes to their interface via simulations with their own customers. Such data could be used in combination with other tools, such as data science techniques, to further enhance predictive accuracy.
• Extending to more complex decision problems: In most online experiments, respondents make their decisions individually and the simulation lasts for a short amount of time (~10-15 minutes). With increased access to potential participants, new types of experiments become possible. For example, individuals can be prompted to make decisions over different periods, which can be used to study how choices change over time. In addition, individuals can be connected to others in real time, which makes it possible to study new decision problems, such as interpersonal trust and

This article grew out of our shared interest in using online labs for evidence-based policy. Janna is Head of Product for Predictiv, the online laboratory platform of BIT, and Jeroen is a behavioural economist at the Financial Conduct Authority and guest lecturer in behavioural science at the London School of Economics. He recently published an FCA paper “Using online experiments for behaviourally informed policy”.