
Alexander Clark
Head of International APAC
“Village doctors” are the primary healthcare providers for many Bangladeshis, but their qualifications and expertise can vary considerably. In the wake of the coronavirus pandemic, the role of village doctors has expanded to pandemic preventers and the advice they provide has become a matter of public health. With that in mind, we partnered with BRAC, the largest NGO in the world, and health-tech start-up Jeeon, to evaluate their flagship ‘village doctor’ training programme in Bangladesh.
The programme endeavours to provide village doctors the skills they need to manage the pandemic. In practice this means training them how to provide the correct medical care for coronavirus patients and ensuring the advice they give is in line with the latest COVID-19 evidence. BIT evaluated two versions of this training programme; an “offline” in-person training session and an online training session accessed through mobile phones, to measure the impact of each on village doctor knowledge. See the offline training below:
Both the online and offline training increased village doctor knowledge equally. Our follow up evaluation was unable to detect a difference in patient outcomes. You can see the knowledge scores collected after training charted below:
Let’s restate the result above in a different way: we tested training interventions, and we got a null result – isn’t that bad?
Null results, where there is no detectable difference between trial conditions, is something that most of us who deal in policy experiments are familiar with. It is certainly tempting to think of them as a disappointment – our interventions “didn’t work” – and that is probably the most common lay interpretation.
However it is not the only interpretation. Actually, a “null” can have several possible explanations. Here are a couple of examples that we’ve sampled from this excellent summary:
However, for evaluating online versus offline training, our null result really is good news. That’s because the fact that there is no difference in knowledge outcomes between the two forms of training suggests that the much cheaper online option is just as good as the expensive in-person training programme. We know this because we worked with Jeeon’s project management and finance teams to construct a cost effectiveness analysis – you can see below that online training costs only ৳ 867 (around £7.40) per village doctor:
Fundamentally, how we interpret our results is as important and nuanced as any other part of the research process. Whether you are cheering in your seat at a whopping effect size or staring hopelessly at a null, it is important to work through all the possible explanations. This is because interpretation of results is a moment where it is very easy for biases to creep in. Take, for example, this adversarial study to test whether eye movement affected memory recall, where two teams of “skeptic” and “proponent” authors collected and analysed the same data on the same protocol and nevertheless produced opposite interpretations.
What do we do to avoid misinterpreting results, positive, negative or neutral? There are a few useful strategies we use at BIT. These include:
In our case, our protocol for evaluation confirmed that in the event of a null we would treat both trainings as being effective and seek alternate measures to inform our recommendation to BRAC and Jeeon. In practice, we spoke to Jeeon about their experiences of both, feedback they’d received, and ended up working with them to carry out the cost-benefit analysis. This led us to the happy conclusion that Jeeon’s training of village doctors helped improve knowledge, and that the online training is an impactful and cost-effective way to do so.
With thanks to both Jeeon and BRAC, both fantastic organisations that are doing incredibly important work in Bangladesh around COVID-19 response and more. If you want to find out more about BIT’s work in Bangladesh – including the cluster RCT on handwashing we are analysing, results due in August – please get in touch!
Design and development by Soapbox.