Inside the BBC’s “innovation incubator”

Published: 23 May 2018

We often describe News Labs as the BBC’s “innovation incubator”. It’s a good description of our role in the corporation, but it doesn’t give potential applicants — or friends and inquisitive family members, for that matter — a clear idea of what our team does every day in the BBC’s Broadcasting House.

So we decided to take a snapshot of a typical day in the office, as seen from the desks of seven of our developers, engineers and producers. Only a few of us specialise in one subject area, so each view is representative of the work that could land on any Labber’s desk during one of our project sprints. At the end of a two-month project cycle, we reshuffle our teams and start prototyping our next set of ideas. These generally fall into two categories: new tools to help journalists produce better content, or new news formats for BBC audiences.

And speaking of potential job applicants — we’re hiring! If this sounds like your dream day in the office, check out our ad for a front-end developer (now closed).

10:00 — Manish, Principal Software Engineer

I’m currently working with my colleague James to help transform the output logs from one of our messenger bots into a form that we can query, to gain insights on reach and usage.

We’re using some of the technologies and services provided by AWS to store and transform our log data, giving us flexibility to search it easily. The logs are currently going into AWS CloudWatch Logs streams and we’re looking to inject these into an S3 bucket using Amazon Kinesis Firehose. This should give us the flexibility to run queries directly against the log data using Amazon Athena without too much overhead. It also frees us from having to build and maintain our own database.

I’m getting to learn about both new technologies and the editorial process.”

I frequently find myself working together with my teammates on similar tasks and this is exactly what I find great about my role looking after our infrastructure. Earlier this week, I was helping Grant to improve one of our Jenkins deployment jobs. Next week, I might be supporting another team who are helping migrate some of our internal hardware between datacentres, I might also be supporting our various projects to make the deployment of prototypes easier, or generally advising on good development practices. All the while, I’m getting to learn about both new technologies and the editorial process, from my teammates.

11:00 — Remi, Junior Software Engineer

I’m finishing up tweaks to the search on Window on the Newsroom (WON) 2, a six-week project that explores how we can help journalists enrich their articles with additional content relating to their story. WON pools data feeds from around the BBC into one accessible and searchable interface.

I just fixed a pesky bug and now users can sort their search results. Now that it’s working, I want to clean up and refactor the code before I create a Git pull request.

As well as using familiar tools like Git, React, AWS and Docker, at Labs we’re encouraged to get involved with fun cutting edge technology such as bots, facial recognition and Alexa — and I’ve had the opportunity to work on all three since joining last autumn. The other Labbers are so supportive, and I can’t believe how much I’ve learnt.

Barack Obama in the oval office. His face is identified by the software — A facial recognition tool, prototyped by Remi and colleagues, during a six-week project sprint

I’ve also had the opportunity to get involved in outreach work. I’ve shown off our work at the BBC’s Design and Engineering Town Halls, written blog posts for our website and also gotten stuck into state-of-the-art tech at MozFest.

12:00 — Grant, Bot Development Producer

I’m working on bots. They’re part of the BBC’s drive to make news accessible to every audience, not just the politically engaged. They’re great because they make it possible for a reader to join a complicated story “halfway through” — but they’re also tricky for journalists, because it’s harder to write an interactive, non-linear story than the traditional 400 or 2000 words, or three-minute TV slot.

So right now, we’re asking questions like “How can journalists get a chatbot online in 15 minutes to help flesh out a popular story?” Or, “If a story is presented as a Q&A, will that stop light readers giving up on complex issues?” To test this, I coded a bot-builder in Rails, whose output you can see in the in-article bot. It’s a team project: the front end was coded by Tom Maslen, an ex-Labs staffer, and the BBC Visual Journalism Team. And James and Luke from Labs have been working on making bot output available in Facebook Messenger.

Chatbot questions about flu — A screenshot of our in-article chatbots, which we developed with the Visual Journalism team

And the future? This afternoon, I met with BBC R&D to discuss how we could centralise the BBC on a single “bot story” data format. Then it should be easy to support bots across all content, all channels and right across the Corporation. But that conversation immediately turned into what, to me, is a much more interesting discussion:

“Can we use bots to make it a very normal, standard thing for an audience to talk back to news organisations? And if we had real two–way conversation with audiences, how could that change news gathering?”

13:00 — Team Lunch 🍪

On sunny days, you might find us in Cavendish Square. On rainy days, we might be in the BBC’s Media Cafe overlooking the newsroom. And regardless of the weather, you’ll almost always find a Labber restocking our biscuit tin or making a round of afternoon teas or coffees for the team.

14:00 — Tamsin, Software Engineer

I’m working on a pilot project experimenting with the new AMP Story format. For the past few weeks we’ve been working closely with journalists from the Russian and domestic services and we’re about to publish our first three stories, after making a few final adjustments.

Phone screen mockups showing various different stories — The BBC’s first three AMP stories — just in time for the Royal Wedding

My colleague Pietro has improvised a simple CMS which converts the data to AMP Story format using some open source software created by Mic News. I’m uploading some final images to a S3 bucket and checking that the videos work for another story we’ll be publishing tomorrow. The project has involved a lot of collaboration with teams in BBC R&D, the World Service, Operations, Media Services, Marketing and Visual Journalism, and everyone’s been very helpful in making it happen, especially with a certain event in Windsor keeping everyone busy.

The BBC’s Mobile & New Formats Editor is a fan of our work:

So @BBCNews & @BBC_News_Labs have been trying out @ampstories.

This is my favourite so far (it’s lovely)- check it out! #RoyalWedding2018 https://t.co/bPPAT8V9IK
— Nathalie Malinarich (@nmalinarich) May 17, 2018

My colleague Lei has received some final analytics tracking labels and added them to the project so we can judge the success of the stories, which was our final blocker — we can now go live! You can read our first three stories:

I email a few stakeholders to let them know the project is now live and the response is very positive. I bump into Nathalie Malinarich, Mobile & New Formats Editor on the stairs and we go through the stories together. She notices that the onward journey links aren’t working, so I head back upstairs to put a fix in place.

15:00 — James, Software Engineer

I’m currently working on taking some of our in-article chatbot formats and exporting them to Messenger, with a focus on engaging underserved audiences. By using Facebook’s Messenger platform, we can also provide daily and weekly updates on the news local to them, and explore new ways of delivering stories through conversational formats.

My colleague Grant has made a separate web app that helps journalists create conversational journeys for our in-article bots. By taking the data output from this app, we can port the responses into a format for Messenger and optimise the look and feel for the platform.

There’s also a few other AWS services we’ve utilised that are brand new to me — such as Kinesis Firehose and Athena, so it’s a great learning experience.”

We’ve gone server-less for this Messenger Bot runner — using AWS Lambda fronted by API Gateway to talk with Facebook. There’s also a few other AWS services we’ve utilised that are brand new to me — such as Kinesis Firehose and Athena, so it’s a great learning experience.

Currently we’re looking how we can push the daily top stories to our subscribers at scale. Facebook’s Broadcast API takes away the complexity of trying to reach hundreds of thousands of users simultaneously. It’s time to integrate this into our Lambda and test to make sure that we can reach the subscribers.

16:00 — Alvin, Front-End Developer

I’m working on improving the user interface of one of our prototypes that is nearing its release date.

Our current prototype is about addressing a series of little pain points that journalists and content producers face when they’re searching for videos and related content in our internal systems.

Behind the scenes: this is a web prototype that consists of a React UI that communicates with a node.js server that acts as an interface to our backend, which is powered by ElasticSearch.

17:00 — Alli, Outreach Producer

I’m currently working on cleaning up a few GitHub repositories from a recent project that we’re now looking to open-source.

I’m usually busy managing News Labs’ social media channels and producing our events, but I had the opportunity to get involved with the Crossing Divides season this sprint. I helped develop an interactive web application with three other Labbers, brushing up on my JavaScript and learning a lot about React over the course of the project. This was my first audience-facing code at the BBC, and so it’s been really rewarding to see how much more I know about clean code now than I did when I first hacked together some of the project’s data parsing scripts. I’m also excited to be the guinea pig for the open-sourcing process, as there’s been a lot of interest around doing more of our coding in the open.

A screenshot of the Crossing Divides web application showing two cartoon faces having a conversation — A screenshot of the Crossing Divides web application

I think this current task really encapsulates the things I love most about my job: the opportunity to use so many different skills, from social media production to news reporting to front-end development, on a day-to-day basis. Everyone in Labs has been extremely supportive of my development as a coder and a journalist, and I often feel that the biggest challenge is focusing on just one thing to do during a project sprint.