Can we automatically generate news comics from text?
Younger audiences are interested in stories with great public service value, such as issues around mental health, but might be less interested in the text-heavy format many of these stories take online. Can we build a new format for telling these stories in a visually appealing way, without significantly increasing journalists' workload?
The Graphical Storytelling project is investigating ways to generate short news comics based on text provided by a journalist, to tell health stories in an accessible and appealing way.
User research with a hand-crafted prototype of the stories we envisage telling was overwhelmingly positive with the younger demographic we hope to reach. Other teams have successfully used similar formats to tell stories using Instagram and Snapchat.
There are significant technical and editorial challenges to overcome. We don't want crafting these stories to be a burdensome task. How can we lean on automation to identify features in text to choose the most appropriate graphical treatments?
Phase One: Can we do this?
In June 2019 we began the first six week phase of the project, to determine if building this would be technically feasible. We studied health stories to identify common structural elements and entities, to better understand the visual elements we'd need to assemble. We built prototypes of entity detection and image lookup services, and a renderer with templates for recurring panel types, such as quotations, statistics, and graphical symbols. We joined these together with a simple web app that let a user input some text, then review and tweak the comic panel that would be generated as a result.
The comics produced at this stage were very simple, but were enough to establish it would be feasible to pursue a more sophisticated way of assembling images. Exploring this prototype, we found it would invite journalists to spend a lot of time tweaking the visual appearance of individual panels, which is not in the spirit of the project. For our second phase, it became clear we need to look at how we can do more of the heavy lifting to choose the best panels automatically.
Phase Two: Getting smarter
In August 2019, we began our second six week project. This time we're focusing on increasing the level of automation in generating comics. We're building a natural language processing pipeline to identify the features in the text entered by a journalist, and generating a resulting "best-fit" image. For example, if we detect numerical data, we will generate a simple data visualisation. If we detect a quotation, we'll show it in a speech bubble with the speaker attributed.
At the same time, we're getting a better understanding of the gaps that automation is unlikely to fill. For example, if a story is describing a novel treatment for an obscure condition, we might not be able to show a relevant graphical representation. Ultimately, we want to close this gap as much as possible. Understanding the size of the gap will help us to understand what other types of panels we need to design, and identify missing graphics to build or possibly commission.
We're also working with our UX colleagues to build an interface that better helps journalists conceive of the whole story as a visually coherent comic, rather than as discrete panels.
- An early prototype is being used to get feedback from potential users from the health news desk.
- We're just getting started, so we'll have more to share about how this project is progressing soon.