20 million job applications passed through the trial – making it one of the biggest experimental social policy trials ever published
With almost 40% of employees working from home this year, remote working and other forms of flexibility have become the norm and myths about productivity losses have started to evaporate.
2020 has given many the opportunity for a more flexible working pattern – and the appetite for flexibility hit new heights during the COVID-19 pandemic. In fact, after experiencing lockdown, 9 out of 10 jobseekers want increased flexibility, be it remote working (60%), flexitime (54%) or reduced hours (26%).
However, this new-found fervour for flexible working has been met with a supply problem in the job market. Even during this pandemic year, just 22% of quality jobs were advertised with flexible working options. Given the increasing demand for flexibility among job seekers, employers who want to attract more talent might consider advertising more jobs as flexible.
Earlier this year, we reported on our field trial with Indeed – the global job site – where we tested a simple nudge to encourage employers to advertise more jobs as flexible. We changed the way options were presented on Indeed’s job advert template. We included a new screen, which prompted employers to choose flexibility options for their vacancy. This led to a 20% increase in the number of jobs advertised as flexible. Moreover, our research also showed that job adverts offering flexible working attracted up to 30% more applicants. If adopted on the Indeed job site alone, we estimated (pre-COVID-19) that, all things being equal, this nudge would add at least 174,000 flexible jobs to the UK economy in a year, and an increased talent pool for employers.
Go big or go home
The replication crisis in social sciences has been well documented. So even though our original sample size was already large, we wanted to make double sure our findings held. Consequently, later in 2019, we ran a second trial – a replication and augmentation of the first one. Here, in addition to the prompted choice screen we added two more arms with behavioural messaging. Taken together, we tested 4 treatments:
- Business-as-usual with no prompt.
- Prompted choice nudging employers to select flexible working options.
- Legal right: On the top of prompted choice, we added a message highlighting that requesting flexible work is a legal right in the UK, based on a previous result showing this can increase uptake of options.
- Gender inclusiveness: On the top of prompted choice, a message flagging that 9 out of 10 women – and men – seek flexible working. The aim was to correct common misconceptions of flexibility being mostly of interest to women.
This second trial involved almost 100,000 employers, posting over 500,000 job ads, and almost 14 million job seeker applications.
Figure 1. Illustration of an additional treatment arm (Legal right)
So did this ‘double nudge’ replicate?
In short, the second trial roll out had an implementation error, however, the results still support the findings of the first trial.
The control group was not initially included, due to an implementation error. Indeed retrospectively provided additional data on unexposed advertisers from the same period, as a comparison group. However, when we did balance checks, we found some imbalances between the treatment arms and this comparison group. This means we cannot be sure that differences between the treatments and the control are attributable to the intervention alone. For instance, this non-random assignment could mean that the intervention arms have a higher number of employers who are in general more likely to advertise flexible work, leading us to overestimate the impact of our intervention. This means that our findings do not withstand the highest robustness standards of RCTs. Instead, the second trial can be interpreted as a quasi-experimental design with weak matching. As such, this evaluation can still shed some light on whether our results from the first trial hold.
Subject to the caveats above, we found that the basic prompt increased the number of jobs advertised as flexible by 17% (vs 20% increase in the first trial, from a lower baseline). However, there were no significant differences between the alternatively worded messages. This suggests that the introduction of a prompt to choose flexible options was a robust enough nudge in itself. As with the first trial, this effect was consistent across all types of flexible working, albeit strongest for flexitime (10 percentage points).
Figure 2. Share of job adverts offering flexible working options (Primary outcome measure)
We were able to compare ‘old’ employers – who took part in the first trial – and ‘new’ employers – those not previously exposed to our prompt. Surprisingly, we found that the effect of our nudge does not seem to diminish over time as old employers were as likely to advertise flexibility – in response to the prompt – as they were the first time around. However, new employers (who made up a smaller proportion of the overall sample) were twice as likely to do so, suggesting that our initial findings may be on the lower bound and the effect could be actually higher.
As for job seekers, we again found that job adverts mentioning flexibility attracted more applicants – this time 19% more (vs 30% in the first trial). Whilst this effect was weaker than in the first trial, the strength and direction suggests flexible jobs consistently attract more applicants.
Figure 3. Impact of offering flexible working in job adverts on number of applicants
Minister for Women & Equalities, Liz Truss, said:
“Our commitment to flexible working is based on our desire to open up employment opportunities to people regardless of their sex or location. The shift for many people to work from home during the pandemic has changed mindsets and now is a chance to seize the opportunity of making flexible working the norm, rather than something employees have to specially request.
“The fact is that for many jobs there are invisible restrictions that hold people back – like the need to live in high-cost accommodation close to the centre of cities or maintain working arrangements that are very hard to combine with family or other responsibilities.
“We now have the chance to break down these barriers and boost opportunities for everyone.”
In our next blog, we will report on a follow-up online study in which we unpack more detail behind this important result, notably about who it affected. But one key result was that both women and men were more likely to shortlist job adverts mentioning flexibility, compared to a full-time offer.
In short, this very large scale replication and extension of our earlier result suggests that:
- prompting employers to be transparent about flexibility can boost the offer of flexible jobs,
- that flexible jobs will attract many more candidates.
For BIT, these trials are truly record breaking: with over 100,000 employers and over 20 million jobseeker applications. In raw numbers, this is BIT’s biggest trial so far – and we think could be the largest social policy trial ever conducted in the UK, and among the largest ever conducted in the world. We’re sure someone has a table somewhere…!
This trial was run by the Gender and Behavioural Insights (GABI) programme which is a collaboration between BIT and our funders, the Government Equalities Office. GABI builds the evidence on what works to improve gender equality in the workplace. Professor Iris Bohnet and colleagues (Harvard Kennedy School) and Assistant Professor Mike Luca (Harvard Business School) advised us on this first round of testing.