Memex - Search Engine for the Dark Web

There have been a number of mentions of Memex over the last couple of weeks. The Wall Street Journal describes Memex as ‘a search engine on steroids. Rather than endless pages of Web links, it returns sophisticated infographics that represent the relationships between Web pages, including many that a Google search would miss’.

Developed in the U.S for the Defense Advanced Research Projects Agency (or Darpa), Memex is a search engine which operates on the world wide web and also on the dark web. Unlike google, it can search for sites which don’t declare their existence. Designed to search for sex and people trafficking rings it can mine data to find links between URL’s, text, numbers, images and geographical co-ordinates, and has been used in sucessful prosecutions in the U.S.

The Memex programme is also set to become available as open source architecture. Michael Peck of Forbes say’s ‘DARPA wants a search engine that can be used by commercial as well as government users. In other words, something that you and I can use. Not that DARPA is doing this to help people find hotter porn or cuter cat videos. The U.S. military and intelligence agencies are overloaded with vast amounts of information gathered by everything from NSA eavesdropping, to video imagery from surveillance drones, to monitoring Chinese and Iranian Web sites. Huge quantities of information, obtained at great expense and sometimes questionable legality, are in digital limbo because there are not enough human analysts to sift through it all. Software that can sift through mounds of data to answer a search engine query is at the top of the government’s to-do list’.

Katherine Noyes of IDG News Service thinks there is a role for Memex in the business world. ‘Stock analysts might use Memex to search the Dark Web for information relevant to the stocks they cover, for example. Wine experts could use it to help them find the latest information in that realm’. On the flip side of developing Memex as an open source are the obvious issues about privacy and potentially offering criminal users of the dark web the opportunity to understand how to avoid being found. Clearly there will be complex issues to iron out before Memex can fully illuminate the dark side of the web.