Mining the gaps: using machine learning to map a million data points from agricultural research from the Global South

This shift in thinking will require major shifts in policy, research, and investment. But where should these investments go? What foundations should be strengthened? Which gaps need filling? What’s working? What’s not?

In order to answer these questions in an informed way, we need to examine the evidence that exists and identify areas where more research is needed.

But this is easier said than done.

The evidence base for agriculture is growing exponentially, and while the wider food systems literature may contain many of the solutions we are seeking, they need to be holistically integrated in order to find those needles in the proverbial haystacks.

Evidence synthesis reports, such as systematic and scoping reviews, provide much needed transparent, rigorous evidence for specific questions. But often, broader questions about the whole of the evidence base need to be answered first, the basic who-what-when-where-and-how that comes before trying to apply a more thorough lens.

State of the evidence reports like this one provides more coverage than what we can achieve in a more focused systematic review—a birds-eye view of the evidence base so that we know where to invest in the future. But until recently we lacked the technology to conduct a landscape scan of the millions of articles that are out there.

Earlier work in this area has suggested that the evidence base we have is not fit for the questions we need and want to ask (Lipper et al., 2020). We need additional efforts to help us understand what the current evidence base has found.

And we also need resources to be designed in such a way that we can seamlessly add new data as it emerges from many partners and independent of the sources from which the data originated.

With the aid of artificial intelligence and machine learning technology, we took a deep dive into more than 1.2 million publications to assess the current landscape of research for the Global South.

The result is a clearer picture of what research has been conducted on small-scale farming and post-production systems from 2000 to the present, and where evidence gaps exist.

This, in turn, highlights potential areas for investment in research and innovation for small-scale farms in the Global South, and provides scope for future research questions.