completed

Datastringer

Datastringer will help you subscribe to data sources, and will notify you when patterns arise or thresholds are broken.

Aims

Datastringer will help you subscribe to data sources, and will notify you when patterns arise or thresholds are broken.

It is more or less the exact equivalent of wire agencies' (AFP/AP/Reuters) local journalists who feed the organisation with news (called wires). Except that here, that's a software living in a dataset, not in Iran, that's going to do the job.

Use DataStringer

Picture yourself as a local journalist. You love local police stories, there’s always something to write about, and your audience cares a lot about them. You get these stories by calling your sources in the force in the morning, and by receiving press releases every once in a while. How fantastic would it be if you could have a little piece of software you could tell what you’re looking for, then let it do the leg-work and alert you when it finds something worth your attention?

That’s what Datastringer aims to do. It was built to be a simple as possible, and very much journalism-oriented. The idea is not new in itself, it’s basically a monitoring tool. However, Datastringer wants to:

  • Simplify the process of monitoring streaming pure-data sources,
  • Give you the freedom to use all JavaScript and Node’s power to mash up several datasets at once,
  • Provide a ready-to-use environment to help you define your alert parameters, as well as encourage you to perform basic data-analysis operations on the monitored datasets before setting the alerts.

See the Datastringer Documentation and explanation here for further info.

What is DataStringer?

As journalists, we're spending an awful amount of time reading the news. Going through the papers, our favourite news websites, our Twitter lists, and our RSS feeds, is part of our morning routine. At work, the familiar ticking of wires coming in never stops.

Some of us now work with data too, considering it a source in itself; a source which can give us tips, leads, and stories, just like our informant working in the Home office.

Summary:

We designed Datastringer in an easy to understand way, so you can hopefully grasp the concept quickly... and get started with your own stringers!

datastringer.js is the main program. It uses two concepts:

Stringer

A stringer is just a javascript function:

  function stringer(parameter1, parameter2, .., parameterN, callback) {
    // with the parameters, the stringer
    //  - fetches the data
    //  - does some process over it
    // if an alert is to be fired, the callback is called with the stringer's
    // name and the reason for the alert.
  }

In Datastringer, each stringer is stored in its own js file, named after what the stringer is about. It must be located at the root of the repository.

For example

Datastringer comes with two builtin stringers, located in crime-stringer.js and local-police-stringer.js.

Going further

You can write your own stringers too! This page will get you started.

Use-case

A use-case is a javascript object that aggregates a stringer file name and a set of parameters for it. With that, one can load the stringer and pass it the set of parameters, plus a callback. It allows to re-use the stringer multiple times with different parameters.

For example

Datastringer stores the use-cases in ~/.local/share/datastringer/stringer_use_cases.json (when you first clone the repo, the file is'nt there: you can write it from scratch or run the wizard, which generate one for you). If you peak inside it you'll see something like that:

  [
    {
      "stringer": "local-police-stringer.js"   // the stringer that will be run
      "parameters": ["leicestershire", "C01"]  // the parameters that it's going to get
    },
    {
      "stringer": "local-police-stringer.js"   // this other use-case uses the same stringer
      "parameters": ["metropolitan", "005EG"]  // but see how the parameters are different in this case
    },

    // other use-cases...
  ]

Going further

The use cases file is very easy to tweak and extend. Go over there to see how you can do that!

The main program just reads the list of use-cases stored in ~/.local/share/datastringer/stringers_use_cases.json, and for each one the concerned stringer is run with the given set of parameters, plus a callback that will send you a mail if it is called. And that's about it!

Again, note that this file is not present by default when you clone the repo. It will be created when you run the wizard for the first time.

Links

Results

  • Our prototype was tried by BBC London, but didn’t prove popular with a wide group of journalists as it required basic coding abilities.

Careers

Love data and code?

We'd like to hear from you.