← Semantechs home The tarka Manual EPUBA4 PDFGitHub
Contents
  1. Introduction

Part I — Setting out

  1. 1What tarka does
  2. 2Installing
  3. 3The Bookshop Trail

Part II — TARQL

  1. 4A first mapping
  2. 5TARQL in depth

Part III — OTTR

  1. 6Templates
  2. 7OTTR over tables
  3. 8bOTTR

Part IV — Data frames

  1. 9Python and Polars

Part V — Checking the transformation

  1. 10Shapes and SHACL
  2. 11SPARQL checks
  3. 12The Ontology Quality Suite

Part VI — In production

  1. 13Into a store
  2. 14A pipeline

Appendices

  1. ACommand reference
  2. BPython reference
  3. CDifferences
  4. DTroubleshooting
  5. EThe example files

10. Shapes and SHACL

A mapping that runs is not yet a mapping that is right. This part of the manual is about knowing: this chapter with SHACL, the next with SPARQL, and the one after with the Ontology Quality Suite. Each finds things the others cannot, and the faulty spreadsheet, data/faulty/bookshops.csv, gives them five mistakes to find:

Row Cell Mistake
Marginalia town wigton: Wigtown misspelt
Borderprint founded c.1972: "about 1972" is not a year
The Bookwyrm staff 0: nobody works there?
Bookbarrow website www.example.org/shops/bookbarrow: no https://
Castle Steps Books second_hand Yes: the column says yes

Map it, and every row maps. tarka run exits with status 0, and writes 33 shops; nothing so far has looked at what they say.

Shapes from the mapping

tarka can say what a mapping makes, as SHACL shapes:

tarka shapes -q tarql/bookshops.rq --base https://example.org/bookshop-trail/shapes# -o out/bookshops.shapes.ttl
shape:BookshopShape a sh:NodeShape ;
    sh:targetClass bs:Bookshop ;
    sh:property
        [ sh:path geo:hasDefaultGeometry ; sh:nodeKind sh:IRI ; sh:class geo:Geometry ; sh:class sf:Point ] ,
        [ sh:path rdfs:label ; sh:datatype <http://www.w3.org/1999/02/22-rdf-syntax-ns#langString> ] ,
        [ sh:path wgs84:lat ; sh:datatype xsd:decimal ] ,
        [ sh:path bs:floorArea ; sh:datatype xsd:decimal ] ,
        [ sh:path bs:founded ; sh:datatype xsd:gYear ] ,
        [ sh:path bs:hasCafe ; sh:nodeKind sh:Literal ] ,
        [ sh:path bs:locatedIn ; sh:nodeKind sh:IRI ] ,
        [ sh:path bs:staffCount ; sh:datatype xsd:integer ] ,
        [ sh:path bs:website ; sh:datatype xsd:anyURI ] ,
        …

Each class the mapping asserts gets a shape, and each property what the mapping binds: a datatype from a cast or STRDT, sh:nodeKind sh:IRI from IRI(), sh:class where the object is always typed. The shapes are honest about TARQL: no property is required (sh:minCount), because any cell may be empty, and bs:hasCafe is only a literal, because the IF may give nothing. From OTTR templates the shapes say more, since a template's signature declares which parameters are mandatory: those properties get sh:minCount 1.

These shapes describe the mapping, not the trail. They catch output that does not match its own mapping (a value of the wrong datatype, a literal where an IRI belongs) and are a starting point for your own.

Validating as you write

--validate checks the output against the mapping's shapes, --shapes FILE against your own (repeatable), and --report FILE writes the SHACL report. shapes/trail-extra.ttl holds the manual's own rules, beyond the course's: events and stock records, and that a shop answers the spreadsheet's yes/no questions:

bt:BookshopExtraShape a sh:NodeShape ;
    sh:targetClass bs:Bookshop ;
    sh:property [ sh:path bs:sellsSecondHand ; sh:minCount 1 ;
                  sh:message "No yes or no for second-hand books: is the cell 'Yes', or empty?" ] ,
        [ sh:path bs:hasCafe ; sh:minCount 1 ;
          sh:message "No yes or no for a cafe: is the cell 'Yes', or empty?" ] ,
        [ sh:path bs:founded ; sh:pattern "^[0-9]{4}$" ;
          sh:message "A founding year is four digits: 'about 1972' belongs in a note, not in the year." ] .

The good spreadsheet passes:

tarka run -q tarql/bookshops.rq -i data/bookshops.csv --validate --shapes shapes/trail-extra.ttl -o out/bookshops-validated.ttl

The faulty one does not:

tarka run -q tarql/bookshops.rq -i data/faulty/bookshops.csv --validate --shapes shapes/trail-extra.ttl \
    --report out/faulty-report.ttl -o out/faulty.ttl
  bt:shop-borderprint bs:founded sh:DatatypeConstraintComponent: "c.1972"^^xsd:gYear
  bt:shop-borderprint bs:founded sh:DatatypeConstraintComponent: "c.1972"^^xsd:gYear
  bt:shop-borderprint bs:founded sh:PatternConstraintComponent: "c.1972"^^xsd:gYear
  bt:shop-castle-steps bs:sellsSecondHand sh:MinCountConstraintComponent
tarka: the output does not conform to the shapes (4 results)

tarka exits with status 3, and the output is written all the same: a file that does not conform is still the file the mapping makes, and a pipeline decides what to do with it. The report, in Turtle, carries the messages:

_:r1 a sh:ValidationResult ;
    sh:focusNode bt:shop-borderprint ;
    sh:resultMessage "A founding year is four digits: 'about 1972' belongs in a note, not in the year." ;
    sh:resultPath bs:founded ;
    sh:resultSeverity sh:Violation ;
    sh:sourceConstraintComponent sh:PatternConstraintComponent ;
    sh:value "c.1972"^^xsd:gYear .

Two of the five, found as the file was written. The year breaks the mapping's own shapes as well as the pattern: twice, because the shop is both a geo:Feature and a bs:Bookshop, and the mapping's shapes give each class its own rule for bs:founded. Yes made the mapping's IF give nothing, so the shop has no bs:sellsSecondHand at all, which only a shape that requires one can notice.

Why a pattern for the year?

STRDT("c.1972", xsd:gYear) writes "c.1972"^^xsd:gYear: an ill-typed literal, its text not in the datatype's lexical space. sh:datatype xsd:gYear rejects it, as above, but only in a validator that checks a gYear's lexical form. SHACL_Engine checks it from 0.3.9 (in tarka 0.3.1) and HOLOS from 0.27.0. Their earlier releases rejected "seven"^^xsd:integer, "2025-02-31"^^xsd:date and "yes"^^xsd:boolean, and let "c.1972"^^xsd:gYear through, as other validators may still. A pattern states the rule outright, whichever engine reads it, and its message says what a year should be. Where a value's form matters, say what the form is.

Validating with the trail's own shapes

The SHACL course ends with one shapes graph for the whole trail, reference/s64-the-whole-trail-in-one-shapes-graph.ttl: the rules the trail's data must keep, written by the people who know the trail. They are the real test. Most are about one node, but some reach further: a shop's bs:locatedIn must be a bs:Settlement, and following bs:within from that settlement must reach exactly one country. tarka's output does not contain the settlements. They are reference data. So s64 must run over the output and the reference data together, which is a job for a store: python/validate.py loads both into a HOLOS store and validates there.

store = holosdb.Store()
for name in ["01-vocabulary.ttl", "02-genres.ttl", "03-places.ttl", "05-people.ttl"]:
    store.load(f"reference/{name}")
for f in files:
    store.load(f)
report = store.validate("reference/s64-the-whole-trail-in-one-shapes-graph.ttl")

Over everything Part II and Part III made, the good spreadsheets give:

python python/validate.py out/bookshops.ttl out/stocks.ttl out/stock.ttl out/events.ttl \
    out/works.ttl out/segments.ttl out/connections.ttl
bookshops.ttl against the mapping's shapes: conforms
everything against s64, with the reference data: 0 violations, 6 warnings
   Warning: …author-kirsty-lammond  …SPARQLConstraintComponent: …author-kirsty-lammond is in a cycle of influence. Three authors are, by design (q34).
   …
   Warning: …event-severn-leaf-2025-04-05  …SPARQLConstraintComponent: …event-severn-leaf-2025-04-05 features …author-perrin-oakes, who had died by then. A memorial, or a mistake?
   Warning: …shop-penwith  …SPARQLConstraintComponent: …shop-penwith cannot be reached on foot from The Inkwell. Two shops cannot, by design.

No violations. The six warnings are the course's own puzzles, seeded in its data on purpose (three authors who influence each other in a circle, an event featuring an author who had died, two shops beyond the footpaths), and tarka's output reproduces them faithfully, because it reproduces the data. SHACL calls a report with any result, even a warning, non-conforming; what a pipeline should stop on is a violation.

Now the faulty bookshops, with the same other files:

python python/validate.py out/faulty.ttl out/stocks.ttl out/stock.ttl out/events.ttl \
    out/works.ttl out/segments.ttl out/connections.ttl
faulty.ttl against the mapping's shapes: does not conform
   …shop-borderprint …schema#founded DatatypeConstraintComponent
   …shop-borderprint …schema#founded DatatypeConstraintComponent
everything against s64, with the reference data: 5 violations, 6 warnings
   Violation: …shop-bookbarrow …schema#website …PatternConstraintComponent: At most one website, with a scheme: {$value}.
   Violation: …shop-bookwyrm …schema#staffCount …MinInclusiveConstraintComponent: Staff count is a positive integer: {$value}.
   Violation: …shop-borderprint …schema#founded …DatatypeConstraintComponent: A shop records the one year it opened, as an xsd:gYear.
   Violation: …shop-marginalia …schema#locatedIn …ClassConstraintComponent: A shop is in exactly one settlement.
   Violation: …shop-marginalia _:… …QualifiedMinCountConstraintComponent: {$this} is in some number of countries other than one.

Four of the five mistakes, in five violations, found by the course's rules: the website without a scheme, the shop with no staff, the year, and wigton, which breaks two rules, since it is not a settlement and so the shop is in no country at all. Only validation with the reference data could find wigton: bt:place-wigton is a perfectly good IRI, and the shapes from the mapping, which know nothing of places, are content. Only Yes gets past s64, which lets a shop say nothing about second-hand books.

Who found what

Mistake Mapping's shapes trail-extra.ttl (tarka --shapes) s64 with reference data
wigton yes: not a settlement, no country
c.1972 yes: datatype yes, by pattern yes: datatype
staff 0 yes: below 1
no https:// yes: pattern
Yes yes: required

No one set of shapes found everything. The mapping's shapes guard the mapping; the project's own shapes guard what the spreadsheets must answer; the domain's shapes, run with the data they refer to, guard the meaning. Use all three: the first two as the file is written (--validate --shapes), the third where the reference data is, in a store.

SHACL from Python, and from the command line

From Python, without a store, SHACL_Engine validates text:

import shacl, tarka_rdf
m = tarka_rdf.Mapping.tarql_file("tarql/bookshops.rq")
report = shacl.Shapes.from_turtle(m.shapes()).validate_turtle(m.write(df))
report.conforms, [(r.focus_node, r.path, r.component) for r in report.results]

and shacl.validate("data.ttl", "shapes.ttl") validates files. SHACL_Engine's own command line, from its releases, does the same in a shell. HOLOS's holos validate validates a store.

The SHACL course teaches every shape in s64, and SemOps Manual, Chapter 8 puts validation into the modelling stage of a semantic practice.