← Semantechs home The tarka Manual EPUBA4 PDFGitHub
Contents
  1. Introduction

Part I — Setting out

  1. 1What tarka does
  2. 2Installing
  3. 3The Bookshop Trail

Part II — TARQL

  1. 4A first mapping
  2. 5TARQL in depth

Part III — OTTR

  1. 6Templates
  2. 7OTTR over tables
  3. 8bOTTR

Part IV — Data frames

  1. 9Python and Polars

Part V — Checking the transformation

  1. 10Shapes and SHACL
  2. 11SPARQL checks
  3. 12The Ontology Quality Suite

Part VI — In production

  1. 13Into a store
  2. 14A pipeline

Appendices

  1. ACommand reference
  2. BPython reference
  3. CDifferences
  4. DTroubleshooting
  5. EThe example files

11. SPARQL checks

SHACL validates nodes against shapes. Some questions are not about one node: do two spreadsheets agree? Did the transformation make what the source data has, no more, no less? Can the graph answer the questions it was built for? Those are SPARQL's, and HOLOS answers them: tools/sparql.py loads tarka's output, with the reference data, into a HOLOS store (holosdb) and runs a query.

First, count

The first thing to look at after any run, and the first to compare between one run and the next (sparql/counts.rq):

SELECT ?class (COUNT(DISTINCT ?thing) AS ?things) WHERE {
  ?thing a ?class .
  FILTER(STRSTARTS(STR(?class), STR(bs:)))
}
GROUP BY ?class
ORDER BY ?class
python tools/sparql.py sparql/counts.rq out/bookshops.ttl out/stocks.ttl out/stock.ttl \
    out/events.ttl out/works.ttl out/segments.ttl out/connections.ttl --no-reference
bs:Bookshop  33
bs:Event  59
bs:StockRecord  95
bs:TrailSegment  33
bs:Translation  6
bs:Work  74
6 rows

Every number is the spreadsheet's row count, and the six translations are the six rows with a translation_of. A count that moves between runs, with no change to the data, is a bug; a count that is wrong on the first run is the cheapest bug you will ever find.

When every file is right and the set is wrong

Each spreadsheet can be internally perfect and the five can still disagree: a title a shop's row lists that stock.csv has no record of, an event at a shop that does not exist, a shop in a town the reference data does not know. No shape over one file can see it, and no shape over one node. sparql/cross-file-checks.rq asks for each, as one SELECT whose every row is a fault:

SELECT ?fault ?subject ?object WHERE {
  {
    # bookshops.csv says a shop stocks a title, but stock.csv has no record of it
    ?subject bs:stocks ?object .
    FILTER NOT EXISTS { ?record bs:atShop ?subject ; bs:ofWork ?object }
    BIND("stocked, but no stock record" AS ?fault)
  } UNION {
    # events.csv names a shop bookshops.csv does not have
    ?event bs:heldAt ?object .
    FILTER NOT EXISTS { ?object a bs:Bookshop }
    BIND(?event AS ?subject)
    BIND("an event at an unknown shop" AS ?fault)
  } UNION {
    # a shop in a place the reference data does not know as a settlement
    ?subject a bs:Bookshop ; bs:locatedIn ?object .
    FILTER NOT EXISTS { ?object a bs:Settlement }
    BIND("a shop in an unknown town" AS ?fault)
  }
  # … and the two other directions: a record the shop's row does not list, a title with no work
}
ORDER BY ?fault ?subject

Over the good spreadsheets' output, with the reference data:

0 rows

Over the faulty bookshops instead:

"a shop in an unknown town"  bt:shop-marginalia  bt:place-wigton
1 rows

The same fault s64 found in Chapter 10, found again, more plainly. "No rows, no faults" is easy to run in a pipeline and easy to read in a report. SemOps Manual §10.6 makes the general case: the set of files is a thing in itself, and must be checked as one.

Asking what shapes allow

The course's shapes let a shop say nothing about a cafe; the committee's spreadsheet asks every shop. When the question is "does any shop fail to answer?", an ASK is the plainest form (sparql/every-shop-answers.rq):

ASK {
  ?shop a bs:Bookshop .
  FILTER (NOT EXISTS { ?shop bs:sellsSecondHand ?any } || NOT EXISTS { ?shop bs:hasCafe ?any })
}
ASK: no      # the good spreadsheet
ASK: yes     # the faulty one: Castle Steps' "Yes"

trail-extra.ttl says the same as a shape. Write a rule as a shape when it is about one node and should be checked as the file is written; as a query when it is about the set, or when the answer you want is a list rather than a report.

The round trip: is it the course's graph?

The strongest check is the simplest to state. The spreadsheets were written from the course's data; mapping them back must give the course's data. tools/compare.py loads what tarka made and the course's graph into one HOLOS store, as two named graphs, and counts three kinds of difference, each a SPARQL query:

# made, not in the course: a triple tarka made that the course does not have
SELECT ?s ?p ?o WHERE { GRAPH <urn:made> { ?s ?p ?o }
  FILTER NOT EXISTS { GRAPH <urn:course> { ?s ?p ?o2 } FILTER (same(?o, ?o2)) } }

and its mirror, in the course, not made, for a value the course gives a subject and property tarka writes; and subjects missed, a thing of a class tarka makes that tarka did not make. "Same" deserves care. tarka writes "80"^^xsd:decimal where the course has "80.0": equal under SPARQL's =, so the comparison uses = for literals. WKT literals are compared as geometries, with geof:sfEquals from HOLOS's GeoSPARQL functions: the course writes POINT(-2.36280 …), tarka POINT(-2.3628 …), and they are the same point.

Everything Part II's TARQL queries made:

python tools/compare.py out/bookshops.ttl out/stocks.ttl out/stock.ttl out/events.ttl out/works.ttl
2472 triples made
made, not in the course: 0
in the course, not made: 0
subjects missed: 0
base direction (RDF 1.2), which tarka does not write: 6
    <…book-the-dark-sea-ar> <http://purl.org/dc/terms/title> "البحر المظلم"@ar--rtl
    …

Everything Part III's OTTR templates and bOTTR maps made, from the Parquet stock table:

python tools/compare.py out/bookshops-bottr.ttl out/shop-stock-parquet.ttl out/segments.ttl out/connections.ttl
1587 triples made
made, not in the course: 0
in the course, not made: 0
subjects missed: 0
base direction (RDF 1.2), which tarka does not write: 0

Every triple the course has for the bookshops, the titles they stock, their stock, their events, the works and the trail, and no other. The six counted apart are the three right-to-left titles of Chapter 5, whose direction RDF 1.1 cannot express.

You will not usually have the target graph to compare with; this manual made its input from one on purpose. But you often have last month's output, and the same comparison, last month's graph against this month's, says exactly what a change to a mapping changed.

Asking the graph its questions

A graph is built to answer questions. The SPARQL course's first, q01, every bookshop on the trail, runs unchanged on tarka's output:

python tools/sparql.py reference/q01-every-bookshop-on-the-trail.rq out/bookshops.ttl --no-reference
bt:shop-bookbarrow  "Bookbarrow"@en
bt:shop-borderprint  "Borderprint"@en
bt:shop-candlemas  "Candlemas Books"@en
…
33 rows

Keep a project's competency questions as .rq files beside its mappings and run them on every build; an answer that changes is a finding. The SPARQL course has seventy more for the trail.

Running queries elsewhere

tools/sparql.py is a few lines of holosdb; the queries are plain SPARQL and run in any store. HOLOS's command line runs them over files with no Python at all, and validates too:

holos query --data out/bookshops.ttl --data reference/03-places.ttl --query-file sparql/cross-file-checks.rq
holos validate --data out/bookshops.ttl --data reference/03-places.ttl --shapes reference/s64-the-whole-trail-in-one-shapes-graph.ttl

Against the server of Chapter 13, any SPARQL client can.