ELMer: What We Learned from Multivariate Testing


Our automated External Links Manager didn’t generate the high click-through rates we hoped for. But it did teach us something about running a multivariate campaign on BBC News story pages.

The BBC is keen to share interesting content from other news providers with our audiences. We were curious to see whether suggesting links to other news sites based on the content our audience members were already reading on the BBC site would encourage them to follow more links. So last month, we set out to test whether our audience benefit from editorially curated links to local news organisations.

Currently, onward journeys are generated through LexisNexis’s Moreover solution, which is based on the geographic location of the story. We used an algorithm developed by BBC R&D and stories ingested by the Juicer to power our External Links Manager (ELMer). Our idea was to analyse the content of BBC stories and try to match that with stories from other news organisations which Juicer thought were similar.

The Trial

We ran our trial on the story pages of two local regions, Birmingham & Black Country and Shropshire. The stories in these regions tend to receive less traffic than national news stories. Because of this, we needed to run the multivariate testing for a longer period of time to build up indicative numbers.

This data allowed us to start exploring our main hypothesis — that the similar stories suggested by our algorithms and then curated by journalists would generate more onward journeys — as well as a few secondary questions:

  1. Would curated ELMer content perform better than LexisNexis’ Moreover content?
  2. Would including articles from across England provide better stories for the journalists to select from and create higher click-throughs than restricting ourselves to a regional recipe?
  3. Does positioning the recommended links element below the story perform better than placing it in the right-hand column for those using desktop machines?

We randomly selected audience members based on the hexadecimal string in their cookie to receive one of six variants:

  1. No components on page
  2. LexisNexis only
  3. ELMer only on the right
  4. Elmer only at the bottom
  5. Both components on the page, at the bottom
  6. Both components on the page, ELMer on the right

We then counted the clicks that each of these components for each of these variants received.

The Results

Would curated ELMer content perform better than LexisNexis’ Moreover (LN) content?
Possibly. While out-and-out performance by ELMer was down against LexisNexis, there was one factor that we couldn’t mitigate. LexisNexis always provides 6 links on story pages, whereas our staff were adding an average of 2-3 links per page, making the totals difficult to directly compare.

Would using a national recipe provide better stories for the journalists to select from and create higher click-throughs than a regional recipe?
There are two ways our content aggregation engine can ‘read’ content. The slow way is to process it, extract concepts and then map those concepts to other related concepts — things like actors and locations. The faster way is to find related content based on word frequency. We chose the second method as it’s fast and therefore has minimal impact on the workflow of journalists using the tool.

In this case, the recipe change didn’t make that much difference. Journalists were discounting many of the stories recommended by ELMer as not meeting the BBC’s guidelines or being too old to be pertinent. The number of ‘valid’ links returned was still about the same either way, and hence the click-throughs didn’t change that significantly.

Does positioning the element below the story perform better than in the right-hand column?
Yes, there were significantly more click-throughs when the elements were presented below the story.

What do we do next?

We are rethinking our approach to finding better ways to link out to other news organisations. We may run another trial for stories that have a greater read time, like feature articles, or change the way in which stories are ingested into the system in the first place.

For all future multivariate tests, we now have a structure and framework in place which works across News Online.