knowledge graphs · SPARQL · SHACL
Show Your Working
A Technical Companion to the Knowledge Graph Articles
At school, a right answer with no working earned half marks at best. The teacher wanted to see how you got there, partly to give credit and partly because the working is where a mistake can be found. The articles in this series have mostly shown answers. Each told a story for readers new to the technology and kept the machinery out of sight: a few sentences of subject, verb and object, a short block of SPARQL, a line saying that two engines had found the same results. This one shows the working.
Behind each illustration was a small file of data and a set of queries or shapes. Some were described in notes; the first three articles had none at all, only the sentences in the text. I have now gathered them all, built out the missing ones, and laid them out as thirty worked examples, numbered W01 to W30. Every one opens in the Turtle Editor Viewer (semantechs.co.uk) with a single link, with its data loaded and, where it needs them, its shapes ready to run. Every one says what result it should give, and every one has been run with the engines the editor uses and checked against that result. The table at the end lists them all.
They are laid out in the same way as the SPARQL course (semantechs.co.uk) and the SHACL course (semantechs.co.uk), which teach the two languages on an invented network of bookshops. The courses explain the languages; these examples show them at work on the problems the articles described. If a construct in an example is new to you, the courses are the place to learn it.
This article is for readers who want to look under the bonnet: data engineers, architects, ontologists and the curious. It summarises what each article modelled and how, draws out the patterns that recur, and reports what running the examples taught me. I should declare an interest once more: I build HOLOS, the triplestore, and the SHACL engine, and both run inside the editor.
How the examples work
Each example is one file, a SPARQL query (.rq) or a shapes graph (.ttl), and each starts with a header in the courses' format:
- what it asks, checks or derives, usually in the article's own words;
- how it works, construct by construct;
- a small diagram of the data involved and the result;
- what to take away;
- a "try it": one change to make, and what should happen when you make it;
- the data it needs, the link that opens it in the editor, the specification it relies on, and the result it should give.
Follow the link and the editor opens with the data in the first tab. For shapes, a second tab holds the shapes graph, already chosen for validation, and where rules are involved the Inference setting is already made. Press Validate. For a query, paste the file into the SPARQL panel and press Execute. Then do the "try it". The point of most examples is the second run: change a source and see what is recalled, extend a certificate and see a violation go, move a sale date into the window and see a holding come into scope.
The data is of two kinds. Where an article used real data, the example uses it with its values unchanged: nine listings from Fáilte Ireland's open data in the tourism examples, and the European Environment Agency's wastewater returns, Eurostat's guest nights and gist's reference data in the circular economy examples. Everything else is invented: the glen, its estate and companies, the lots of oats, the journey's times, the lets and boats, the inquiry. The invented data is not arbitrary, though. It is built so that each example has something to find. Five holdings around the glen each fail one clause of a question, except one that meets them all. Each set of records breaks each rule once. A test that can only pass proves very little.
A script in the collection (scripts/check.mjs) runs every example in Node with the same WebAssembly builds of HOLOS (0.26.3) and the SHACL engine (0.3.7) that the editor runs, and checks each against its header, "try it" included. All thirty examples give the results their headers state, and so do the twenty-one "try it" changes that edit the data or the query; the other nine ask you to read something in the result.
Nine articles, thirty examples
Joining the Dots (W01 to W04)
The first article (Joining the Dots) introduced the toolkit through investigative journalism, framed as a supply chain: sources as raw material, sentences as components, shapes as goods-in inspection, rules as assembly, and provenance as the bill of materials. It had three sentences, about Acme Holdings, Jane Doe and a flat on Harbour Street, and no data behind them. The examples build a small investigation around those sentences: six companies, seven people, four meetings, two payments, and the sources each claim came from.
W01 is the article's question (a director of a company registered in Cyprus that owns a property in London, who has met a minister in the last two years) written as SPARQL clause by clause. The data has two people called John Smith: one a director, the other a lobbyist who met the minister. Joined by identifier, neither is in the answer. The "try it" joins them by name instead and produces a false lead, which is the article's point about moving from strings to things. W02 is goods-in inspection: a payment with no currency, a claim with no source, a directorship that ends before it starts. W03 writes the article's two Datalog rules (more than half means control; control passes along a chain) as SHACL-AF rules, and follows control through three layers of companies to a sanctioned trust. W04 is the recall question: if the leaked register is forged, which claims, arguments and stories rest on it?
The directorships are written in RDF 1.2. The triple is asserted, and a reifier carries its source and dates: the "statements about statements" the article said were coming. The shareholdings are ordinary nodes, because a percentage belongs to the holding rather than to either party. The examples use both on purpose, and the shapes treat them alike.
Who Owns the Glen? (W05 to W07)
The land article (Who Owns the Glen?) introduced the invented glen of Glenmeadhanach, its estate company registered in Jersey, and the trust behind the company. The examples give the glen five holdings, seven titles, nine parcels, eight proprietorships with dates, four controlled interests, and a community body with a registered interest in land.
W05 is the article's question with a map attached: holdings over 1,000 hectares that have changed hands in the last five years, lie within ten kilometres of a community with a registered interest, and are ultimately controlled from outside the UK. It uses GeoSPARQL's distance function, a path to find the top of a chain of control, and a negation over transitive containment to decide what counts as inside the UK. Each of the other four holdings fails exactly one clause. W06 checks the article's list: a title with no proprietor recorded, a parcel with no boundary, a holding whose stated area is 18% more than its parcels add up to, and two sole owners whose dates overlap. W07 applies the article's rule, that a controlled interest in a company that owns a title is an interest in the title, through every layer there is.
Field to Fork and Back Again (W08 to W11)
The food article (Field to Fork and Back Again) followed a packet of porridge back to the field and the milk in it back to the farms, and introduced gist (github.com), Semantic Arts' small upper ontology, which the rest of the series uses. Its distinctions matter in the examples.
W08 asks which products contain anything grown in a river catchment. No record says so. The answer comes from gist:isGeoContainedIn, which gist defines as transitive, followed upwards from the field, and from a path down through what each product is made of. W10 asks which farm a batch of cheddar came from, and gets the article's answer: no single farm, but five farms in proportions, because gist:isMadeUpOf, for substances, is not gist:isPartOf, for parts. W09 is goods-in again, with the open world at its centre: a product that says nothing about sesame has not said it contains none, so the shape asks for an explicit "contains" or "free from" for every allergen. W11 is the article's coeliac rule, with its exception for certified gluten-free products written into the rule where anyone can see it.
The Wood and the Trees (W12)
The forestry article (The Wood and the Trees) used one shapes graph for certification: record-level checks on compartments and felling permissions, and UKWAS's requirement that 15% of a woodland unit be managed for biodiversity. W12 is the article's own example. The unit falls just short, at 14.8%, and one permission has no restocking date. The instructive part is the split. A record-level rule looks at one compartment and can run on every change; the 15% rule needs the whole unit and belongs at audit.
Down the Line (W13 and W14)
The electricity article's examples (Down the Line) are its own. W13 finds the homes within 500 metres of a new 400 kV line with geof:buffer and geof:sfWithin. W14 is the connections reform's land test in SHACL Core: secured land rights mean an option, ownership or a long enough lease, which is sh:or over three shapes. The ownership in W14 cites title QED00001 by its number, as a string. In W05 to W07 the same title is a node with an identifier of its own. A federation would make the two the same thing, and that join is the subject of the proposals at the end of each article.
Across the Water (W15 and W16)
The travel article (Across the Water) followed nine legs from the glen to Dublin. W15 checks every connection, using the actual arrival where one is recorded and the planned arrival otherwise, and finds the two that fail: three minutes to change trains in Glasgow, and a ferry check-in missed because a coach was held up on the A77. W16 lists the stops with their names in English, Gaelic and Irish, straight from language-tagged values.
Off the Beaten Track (W17 and W18)
The tourism article (Off the Beaten Track) combined nine real Fáilte Ireland listings with an invented layer of lets, a register and boats to Skellig Michael. W17 finds fourteen problems, nine of them in the real data: none of Fáilte Ireland's names says which language it is in. W18 is the "open on the day" query, and it has a story of its own, which I come to below.
Ebb and Flow (W19 to W22)
The circular economy article (Ebb and Flow) took Dingle's treatment plant and its busiest week, with real figures from Ireland's 2022 return to the EU and from Eurostat, measured in units from gist's reference data with the population equivalent added. W19 has the article's five shapes and its ten results, including the two official capacities that disagree. W20 is the model of a gentler season, W21 finds flows whose busy seasons cancel each other out, and W22 converts every figure into kilograms of oxygen demand a day using nothing but the reference data's conversion factors.
Chapter and Verse (W23 to W30)
The ninth article (Chapter and Verse), on family history and public inquiries, makes the fullest use of RDF 1.2 in the series. Each source's claim is an unasserted triple term carrying its source; the researcher's conclusion is asserted and points to the claims it weighed. Its eight examples check the genealogist's citation rule, find where sources disagree, and ask the recall question of a family tree. For an invented inquiry, they ask who knew what and when, what had been said by a given day, which statements contradict each other, and which questions remain open.
Patterns across the examples
Read across the thirty and a few ideas carry most of the weight.
Paths do the joining. Many of the queries turn on a property path with + or *. gist:isGeoContainedIn+ turns "is in" into a question at any scale (W08), and following a thing's location first does the same for a flat or a house (W01). prov:wasDerivedFrom* is the recall question (W04, W25). A path up through controlled interests, with a negation to find the controller nobody controls, gives the top of a chain (W05). A path through what a product is made of walks a recipe to any depth (W08, W11). None of these needs to know how long the chain is, which is what makes them useful on real ownership structures and real supply chains.
Relationships with something to say. A directorship has dates and a source; a proprietorship has dates and says whether it is sole; a claim has a stance. The examples use two ways of saying things about a relationship. An ordinary node (a proprietorship, a shareholding, a tanker collection) suits a relationship that is a thing in its own right, with properties of its own. An RDF 1.2 reifier suits a statement you want to keep as a statement, either asserted with its provenance, like the directorships in W01 to W04, or not asserted at all, as a claim, like the genealogy examples. SHACL does not mind which: a reifier is a node, and a shape targets it like any other.
The open world, closed on purpose. A knowledge graph does not assume that what it fails to say is false. The examples keep running into this, and each time a shape is where the assumption is closed, for a purpose. An allergen must be declared either way (W09). A missing proprietor is reported as "not recorded", not "ownerless" (W06). An inquiry's search that found nothing becomes "unknown", never "no" (W26). A place with no published season is "not known", not closed (W18).
Rules are policy, written down. W03, W07 and W11 are rules, and each embodies a choice someone could argue with: that more than half means control, that "owns" means "is the current proprietor of", that certified gluten-free oats are an exception. Written as SHACL-AF rules, each choice sits in one place, and each inferred statement can be traced to it. W03 also makes a practical point. The specification defines one pass of the rules, so a rule that follows a chain stops a step short unless the engine repeats it. The editor offers both.
Arithmetic is a portability problem. Shapes that compare quantities are where engines differ most. The examples keep their arithmetic in whole numbers until a final division: twenty times the difference compared with the stated area, rather than a ratio compared with 0.05. Every engine gets the same answer that way.
Units and places are data. The circular economy examples check each unit against gist's reference data: a unit must belong to a group that the measured aspect allows, so a count of people in tonnes is caught (W19), and converting between units is a join and a multiplication (W22). Geography is the same. A boundary is a value on a node, and distance is a function in a filter (W05, W13).
One upper ontology, used as it defines itself. Every gist class and property in the examples has been checked against gist 14.1.0: each exists, and each is used as gist defines it. The check corrected two of the articles. gist relates one geographic location to another with gist:isGeoContainedIn, and a physical thing to its location with gist:hasPhysicalLocation. So a house in the glen (W13), a flat in London (W01) or a pier in Dún Chaoin (W17, W18) has a location, and the location lies within a region; the house does not lie within the region directly. And gist 14.1 retires its class for landmarks and asks for one to be defined under its class for physical things, which the tourism examples now do. The questions do not change: a path that follows the location and then the regions (gist:hasPhysicalLocation/gist:isGeoContainedIn*) finds the same flat in London.
One identifier, many examples. Glenmeadhanach Estates Ltd is the proprietor of two titles in W05, owns the estate the oats come from in W08, and supplies milk in W10. Title QED00001 appears in W05 and W14. The examples share a namespace for the glen, so the company is the same identifier in each. They do not share everything yet: W14 names the title as a string. That is a small version of the problem every article described, and of its solution.
What the engines taught me
Running every example through the browser builds of both engines turned up a fault worth reporting, and it has been fixed.
Until version 0.26.3, HOLOS could not compare two xsd:date values, and the SHACL engine's SPARQL constraints could not until 0.3.7. A comparison such as "the day is on or before the end of the season", made between two dates, was an error, and a filter that errors removes the row. Equality worked, sorting worked, and xsd:dateTime values compared, which is why it went unnoticed. The tourism article's "open on the day" query compared dates in exactly that way. Run for 21 November, the article's day, it said that the Blasket Centre and Skellig were closed, which was true. Run for 15 July, it said they were closed too, which was not. In November the error and the truth happened to agree, because both places were shut. The cause was a feature of the Oxigraph libraries both engines build on, which gives dates, times and durations their values only when it is switched on. It is now on in both, and with it comes arithmetic: one date subtracted from another gives a duration. Five of the examples (W01, W05, W06, W09, W18) depend on the fix, and their checks fail on the older engines. There is a lesson in it that I should have applied earlier: test a rule about dates on a day when it should say yes.
The same habit had found an earlier fault while the circular economy article was being written: an operation on decimals whose exact result needed more than 18 places returned nothing instead of being rounded. That was fixed in HOLOS 0.26.2 and the SHACL engine 0.3.6. The shapes keep to whole numbers anyway, so that every engine gives the same answer to the last digit. And SHACL-AF rules run once, as the specification says, with repetition as a setting of the engine.
Why report faults in my own engines in an article? Because the series has argued that a claim should carry its evidence, and a demonstration is a kind of claim. Every example here says what it should return, on which engines, and how to check. When one did not, the header and the script said so, and it was the engine that changed.
Where next
The examples are small on purpose: each fits on a screen and makes one point. Real work is bigger, and the proposals at the end of each article describe what bigger looks like: holons of named graphs, each kept by whoever owns the facts, with shapes at their boundaries and projections for each audience. Seen together, the thirty examples are a sketch of that. They are nine small graphs with a shared upper ontology and shared identifiers where they overlap, with rules and shapes kept beside the data they govern, and every result traceable to a source.
If you work with data like any of these, try the examples on a copy of your own records: change the namespace, load your data, and see what the shapes find. If something fails that should pass, or passes that should fail, I would like to hear about it.
The examples
Each example opens in the editor from the "open" link. The files are in a demos folder beside this article, one subfolder per article with the data each needs; its README (demos/README.md) explains how to run them from the command line as well.
| Example | Article | Kind | Result | ||
|---|---|---|---|---|---|
| W01 | A director, a company in Cyprus, a flat in London and a minister | Joining the Dots | query | 1 row | open |
| W02 | Checking the goods coming in | Joining the Dots | shapes | 3 violations | open |
| W03 | Control through any number of shell companies | Joining the Dots | rules, to a fixpoint | 3 warnings, 6 inferences | open |
| W04 | The recall question | Joining the Dots | query | 5 rows | open |
| W05 | Holdings in scope: a question with a map attached | Who Owns the Glen? | query | 1 row | open |
| W06 | Inspecting the records | Who Owns the Glen? | shapes | 3 violations, 1 warning | open |
| W07 | Who stands behind a piece of ground | Who Owns the Glen? | rules, to a fixpoint | 6 inferences | open |
| W08 | Everything from the catchment | Field to Fork | query | 2 rows | open |
| W09 | Checking the goods coming in, and the open world | Field to Fork | shapes | 3 violations | open |
| W10 | Which farm did this cheese come from? | Field to Fork | query | 5 rows | open |
| W11 | A rule anyone can argue with | Field to Fork | rules, one pass | 2 inferences | open |
| W12 | Certification as shapes | The Wood and the Trees | shapes | 2 violations | open |
| W13 | Homes within 500 metres of a new line | Down the Line | query | 2 rows | open |
| W14 | Gate 2: secured land rights | Down the Line | shapes | 1 violation | open |
| W15 | Connections as shapes | Across the Water | shapes | 2 violations | open |
| W16 | Names along the way | Across the Water | query | 9 rows | open |
| W17 | Seasons and limits as shapes | Off the Beaten Track | shapes | 14 violations | open |
| W18 | Open on the day | Off the Beaten Track | query | 9 rows | open |
| W19 | Peaks as shapes | Ebb and Flow | shapes | 8 violations, 2 warnings | open |
| W20 | The peak, and the peak spread out | Ebb and Flow | query | 12 rows | open |
| W21 | Flows that are out of step | Ebb and Flow | query | 1 row | open |
| W22 | In other units | Ebb and Flow | query | 12 rows | open |
| W23 | The citation rule as a shape | Chapter and Verse | shapes | conforms | open |
| W24 | Where the sources disagree | Chapter and Verse | query | 2 rows | open |
| W25 | The recall question, for a family tree | Chapter and Verse | query | 2 rows | open |
| W26 | Stance, standing and a protected witness | Chapter and Verse | shapes | conforms | open |
| W27 | Who knew what, and when | Chapter and Verse | query | 5 rows | open |
| W28 | What had been said by a given day | Chapter and Verse | query | 2 rows | open |
| W29 | Contradictions, both sides kept | Chapter and Verse | query | 1 row | open |
| W30 | Open questions | Chapter and Verse | query | 1 row | open |
Back to the classroom
The teacher was right. An answer about land, food or a journey is only as good as the working behind it, and the working is where a mistake can be found: a date compared the wrong way, a rule that stops a step short, a name taken for a person. A knowledge graph keeps the working, in the sources, the rules, the checks and the results, in a form anyone can open and run. Thirty examples are a small start, and each one is a link away.