Joining the Dots

Anyone who has watched a detective drama knows the scene. A wall covered in photographs, newspaper cuttings and index cards, with red string running from pin to pin. The detective stands back, looks at it for a while, and then sees the connection that cracks the case. It is a television cliché, but it is a cliché because it is true to how investigation works. The cards matter, but the string matters more.

Most investigative newsrooms and think tanks have their own version of that wall. These days it is less likely to be cork and string and more likely to be a spreadsheet with too many tabs, a shared drive full of PDFs, a timeline in a word processor, and one researcher who "just knows" that the director of one company is the brother-in-law of a minister's special adviser. The cards are all there. The string, all too often, lives in somebody's head.

This article is about a family of technologies that are designed to keep the string. They go under the general name of knowledge graphs, and they come with a set of acronyms that can be off-putting: RDF, SPARQL, SHACL, OWL, Datalog. I want to explain, without assuming any technical background, what each of these does, and why I think they are particularly well suited to people whose job is to follow connections and then stand behind their conclusions in public.

A supply chain for stories

Before getting to the technology, it helps to look at the work itself from a slightly unusual angle: as manufacturing.

A manufacturer sources raw materials from suppliers, inspects them as they come in, and turns them into standard parts, each with a part number in a catalogue. Parts are assembled into sub-assemblies, and sub-assemblies into a finished product. Alongside all of this sits a document called the bill of materials, which lists every part that went into the product and where each one came from. Nobody outside the factory ever sees it, but it is what lets the manufacturer answer the awkward questions. What is this made of? Who supplied that? And, when a supplier turns out to have shipped a faulty batch, which of our products contain it?

Investigative and policy work has exactly the same shape. Ideas and material are sourced: a tip-off, a leaked spreadsheet, a company filing, a freedom of information response, an interview, an academic paper, a remark over coffee at a conference. They are developed: checked, researched, broken down into individual facts. Those facts are marshalled into what I would call a lexicon, an organised body of the people, organisations, places and events involved, and the relationships between them. From the lexicon, arguments are assembled: this person controls that company, that company won this contract, and therefore there is a conflict of interest. And from the arguments come the finished products, the story, the briefing, the report.

Most organisations take great care over the finished product. There are editors, lawyers and fact-checkers. But the bill of materials, the record of which facts each argument rests on and where each fact came from, usually lives in notebooks, inboxes and memories. That works, right up until the day someone asks "how do you know that?" about a story published two years ago by a reporter who has since moved on.

The technologies in this article map surprisingly neatly onto that supply chain, and the rest of the article follows it from raw material to finished story.

Sentences, not spreadsheets

Let's start with the parts. A spreadsheet asks you to decide your columns up front. That works well for things that look alike, such as a list of payments, but investigations are rarely so tidy. One source tells you about a company's directors, another about its shareholders, a third about a property it owns in another country, and a fourth about a meeting its chairman had with a government department. Each new source wants a new tab, and the connections between the tabs are kept by hand.

RDF, the Resource Description Framework, takes a different approach. Everything is recorded as a short sentence with three parts: a subject, a verb and an object.

Acme Holdings — is registered in —> Cyprus

Jane Doe — is a director of —> Acme Holdings

Acme Holdings — owns —> Flat 3, 12 Harbour Street

Each of these sentences (the technical term is a triple, for obvious reasons) stands on its own. In supply-chain terms they are the components: small, standard and interchangeable. A new source does not need a new table; it just adds more sentences. And because Acme Holdings is the same thing in all three sentences, they join up by themselves. Put enough of them together and you have a web of connected statements, which is all a knowledge graph really is. The string, in other words, is part of the data.

There is one subtlety that matters a great deal to investigators. In a knowledge graph, each thing is given its own unique identifier, rather than being known by its name, a move often summed up as going from "strings to things". Think of it as a part number. A name is just a label, and labels are slippery. "J. Doe" in a leaked document and "Jane Doe" in a company register might be the same person, or they might not. Two people called John Smith are certainly not the same person, however much a spreadsheet lookup might like them to be. By giving each person, company and address its own identifier, the graph lets you say "these two records are the same person" as a deliberate act, which you can source, review and, if necessary, undo. Anyone who has had to issue a correction after confusing two people with the same name will see the value in that.

Agreeing what the words mean

So far so good, but the sentences are only as useful as the agreement behind their verbs. If one researcher records "owns" for a 5% shareholding and another records it only for outright ownership, the graph will cheerfully join the two together and give you a wrong answer with great confidence. A factory that let every workshop invent its own part numbers would have the same problem.

This is where an ontology comes in. An ontology is the agreed catalogue of the kinds of things you care about (people, companies, contracts, meetings, donations) and the relationships between them, each with a definition that the whole team signs up to. It is the backbone of the lexicon. What exactly do we mean by "beneficial owner"? Is a loan a kind of donation? When does a meeting count as lobbying? A newsroom already has a style guide; an ontology is a style guide for the data.

Is this too technical for a newsroom or a think tank? Not really. I have argued before (semanticarts.com) that building an ontology is as much a social process as a technical one, and a good way of finding out where a team does not actually agree. Far better to discover that two colleagues mean different things by "control" during a morning's discussion than in a lawyer's letter after publication.

The good news is that you rarely have to start from a blank sheet. The investigative community has already done a lot of this work. FollowTheMoney (github.com), the data model behind OCCRP's Aleph platform and OpenSanctions (opensanctions.org), describes people, companies, assets, court cases and the relationships and events between them. It is an ontology in all but name. Open Ownership has published an RDF vocabulary (openownership.org) for its Beneficial Ownership Data Standard, and GLEIF, the body behind the Legal Entity Identifier used in financial reporting, publishes ontologies for "who is who" and "who owns whom" (gleif.org). Reusing these is the equivalent of buying standard parts: your data lines up with other people's from the start.

Asking questions of the graph

Once you have a graph, you need somewhere to keep it and a way to ask it questions. The somewhere is called a triplestore, which is simply a database built for these three-part sentences; think of it as the parts store. The way of asking questions is a language called SPARQL (pronounced "sparkle", which tells you something about the people who name these things).

A SPARQL question is a pattern with gaps in it. You describe the shape of what you are looking for, and the triplestore finds every way of filling in the gaps. In plain English, a question might read:

Find any person who is a director of a company registered in Cyprus, where that company owns a property in London, and that person has met a government minister in the last two years.

Asked of a well-built graph, that question takes seconds. Asked of a shared drive, it takes a researcher several days, and depends on that researcher thinking to ask it in the first place.

You do not have to build your own graph to try this. Wikidata, the structured sister project of Wikipedia, is a public knowledge graph with a SPARQL query service (query.wikidata.org) that anyone can use. It holds, among a great many other things, who has held which public office and between which dates. And because SPARQL is a standard, a question can reach across your own private graph and a public one like Wikidata at the same time. Your sensitive material stays where it is, and the public context comes to it.

Checking the goods coming in

Every good newsroom has someone whose job is to ask awkward questions before a story goes out. Where is the source for this? Are these dates the right way round? Is this the right company number? In a factory this is goods-in inspection, and it happens before anything is built, not after. SHACL, the Shapes Constraint Language, is that inspection written down in a form a computer can apply to the whole graph at once.

A SHACL rule might say that every payment must have a date, an amount and a currency; that every claim must point to the document it came from; or that a directorship cannot end before it starts. Run the checks, and you get back a report listing every record that falls short. It will not tell you whether a fact is true. That remains a job for people. But it will tell you, very quickly, where your evidence is incomplete or contradicts itself, which is exactly where you want to look before anyone else does.

There is a related idea that investigators tend to grasp more quickly than most software developers. Knowledge graphs work on what is called an open world assumption. If the graph does not say that Jane Doe was a director of a particular company, that does not mean she wasn't; it only means we have not recorded it. Absence of evidence is not evidence of absence, as every good reporter knows. SHACL is where you get to say, for a particular purpose, "this must be present, and I want to know if it isn't".

Assembling arguments

Now for assembly. As well as storing statements, a knowledge graph can apply rules that work out new statements from the ones it already has. This is the part of the toolkit I find most interesting, and the one least used outside specialist circles.

Rules of this kind are often written in Datalog, a small rules language that has been around since the 1980s. A Datalog rule is a simple "if this, then that", but the engine keeps applying the rules until nothing new turns up, which means they follow chains as far as the chains go. Take this:

If A owns more than half of B, then A controls B.

If A controls B, and B controls C, then A controls C.

Two lines, and the graph will now trace control through any number of shell companies, however many layers the lawyers have stacked up.

This is not an academic example. The US Treasury's "50 Percent Rule" (ofac.treasury.gov) says that a company owned 50% or more, directly or indirectly, by one or more sanctioned people is itself treated as sanctioned, even if it is not on the list itself. There is no public register of such companies, so somebody has to work them out from the ownership data. That is a rule, applied to a graph, and it is exactly the kind of thing a reasoning engine does well and a person with a highlighter pen does slowly.

Just as important, a rules engine can explain itself. Every conclusion it draws can be traced back to the statements and the rule that produced it, which makes each conclusion a sub-assembly with its own parts list. When an editor, a lawyer or a journalist's subject asks "how do you know that?", you can show the chain. Yes, it is all done by computers. But the output is a piece of reasoning a person can read, check and defend, and reuse in the next story that needs it.

Who said so, and when? Keeping the bill of materials

Investigations rarely deal in plain facts. They deal in claims: what the 2019 filing says, what the leaked register says, what the minister's office said on Tuesday and then rather differently on Thursday. A tool that only records "Jane Doe is a director of Acme Holdings" has thrown away the most important part of the story, which is who said so, and when it was true.

Knowledge graphs can keep both. Each source's statements can be kept in its own compartment (a named graph, in the jargon), so you can always ask where a statement came from and compare what different sources say about the same thing. A directorship can carry the dates it started and ended. The next version of the RDF standard, now close to completion at the W3C, goes further and lets you make statements about statements directly, so that "according to the leaked register, Jane Doe was a director from 2015 to 2019" can be recorded as exactly that. OpenSanctions already works in a similar way, recording each value with the source it came from and when it was first and last seen (opensanctions.org).

Put those together and you have the bill of materials. The W3C even has a standard vocabulary for it, called "PROV" (w3.org), which records what each thing was derived from, what activity produced it, and who was responsible. A published story points to the arguments it makes, each argument to the facts it rests on, and each fact to the documents and sources it came from. The same can be done for ideas as well as facts: where a line of inquiry started, who developed it, and which pieces of work it eventually fed. For a think tank, whose ideas are its product, that is a record worth having.

The payoff is the manufacturer's recall question, turned round for investigators. If a source turns out to be unreliable, or a dataset is corrected, or a document proves to be forged, you can ask the graph which facts came from it, which arguments used those facts, and which published pieces relied on those arguments. You can also ask "what did we know on the day we published?" and "which of our sources disagree about this?" Those are questions an investigations editor asks all the time, and they are surprisingly hard to answer from a folder of documents.

One caution, which journalists will already have spotted. A complete bill of materials for a story includes its confidential sources, and must be protected accordingly. The graph can refer to a source by a code name, with the real identity held somewhere else entirely, or nowhere at all. The structure of the reasoning can be kept without the secret.

Hasn't this been done already?

Some of it has. The best-known example is the Panama Papers, where the International Consortium of Investigative Journalists put the leaked records into a graph database and gave reporters around the world a visual tool (neo4j.com) for exploring the connections. Most of those reporters never wrote a query; they clicked from person to company to address and followed the string. That project proved, very publicly, that the connections are where the stories are.

Now, I'm not saying that the particular technologies I have described are the only way to do this. ICIJ used a different style of graph database, and very successfully. What the RDF family adds is that everything is a published standard, not a feature of one product. The identifiers, the vocabularies, the questions, the quality checks, the rules and the bill of materials can all be read by other people's software. That matters when you want to pool data with partner organisations across borders, hand your evidence to a regulator, or publish it so that others can check your working. For a think tank in particular, publishing the evidence behind a report in a form others can query is a strong answer to the charge that you simply chose the facts that suited you.

It also matters in a world where every vendor is offering an AI assistant for investigations. Those tools are fluent but forgetful, and they do not always know where their answers came from. A knowledge graph, with its sources and its rules, is a record of what you actually know and how you know it. If you are being sold an AI tool for this kind of work, it is worth asking what it is grounded on, and whether it can show you its bill of materials.

Getting started

None of this requires a large budget. Most of the tools are free and open source, and a modest graph covering a single investigation will run quite happily on an ordinary laptop. That has a security benefit too: material from a confidential source never needs to leave your own machines.

My advice to a team starting out would be to begin with the finished product and work backwards. Write down the five questions you wish you could ask of your material, and let those decide what goes into the graph. Reuse an existing vocabulary such as FollowTheMoney or the beneficial ownership standard rather than inventing your own. Start with one investigation and a few hundred statements, not your whole archive. And find one technically minded partner, inside or outside the organisation, to help with the plumbing, while the journalists and researchers keep hold of the meaning, which is where their expertise lies.

Be clear about the limits, too. A knowledge graph will not tell you whether a source is lying. Deciding that two records refer to the same person is an editorial judgement, and should be recorded and reviewed as one. And a graph built from poor material will give you poor answers more quickly than ever.

Back to the wall

The detective's wall is, when you think about it, a knowledge graph drawn by hand: things on cards, connections in string, and a person standing back to reason about what it all means. What it lacks is a bill of materials. Take the cards down at the end of the case and the reasoning goes with them.

The technologies I have described do not replace the person standing back from the wall. What they do is keep the string from falling off, make sure everyone on the team means the same thing by each piece of it, check that every card has a source, and keep a record of how each conclusion was assembled and what it was assembled from. Every published story has a bill of materials, whether or not anybody wrote it down. Knowledge graphs are simply a very good way of writing it down.