← Semantechs home The tarka Manual EPUBA4 PDFGitHub
Contents
  1. Introduction

Part I — Setting out

  1. 1What tarka does
  2. 2Installing
  3. 3The Bookshop Trail

Part II — TARQL

  1. 4A first mapping
  2. 5TARQL in depth

Part III — OTTR

  1. 6Templates
  2. 7OTTR over tables
  3. 8bOTTR

Part IV — Data frames

  1. 9Python and Polars

Part V — Checking the transformation

  1. 10Shapes and SHACL
  2. 11SPARQL checks
  3. 12The Ontology Quality Suite

Part VI — In production

  1. 13Into a store
  2. 14A pipeline

Appendices

  1. ACommand reference
  2. BPython reference
  3. CDifferences
  4. DTroubleshooting
  5. EThe example files

9. Python and Polars

Spreadsheets are rarely ready to map. Columns need joining, splitting, gathering into lists, renaming; values need reformatting. In a pipeline that work belongs to a table tool, and in Python the table tool is Polars. tarka's Python package, tarka-rdf, takes a Polars data frame wherever the command takes a file, so the preparing and the mapping happen in one process, without a CSV file between them.

This chapter does both: it walks through python/prepare.py, which made the prepared tables Part III used, and python/frames.py, which maps frames and a Parquet file from Python.

Preparing tables with Polars

OTTR templates read values as they should come out, so the prepared tables hold prefixed names, WKT literals with their datatype, and lists. Every column is read as text, as the committee typed it:

read = lambda name: pl.read_csv(DATA / name, infer_schema=False)

Prefixed names and WKT, per row. An expression per column, all evaluated together:

def prefixed(prefix: str, column: str) -> pl.Expr:
    """`inkwell` -> `shop:inkwell`; an empty cell stays empty."""
    return pl.when(pl.col(column).is_not_null()).then(pl.lit(prefix + ":") + pl.col(column))

shops = read("bookshops.csv").with_columns(
    prefixed("shop", "shop").alias("shop_iri"),
    prefixed("place", "town").alias("town"),
    prefixed("genre", "specialism").alias("specialism"),
    (pl.lit("geom:shop-") + pl.col("shop")).alias("point"),
    # OTTR has no type for WKT, so the cell carries its datatype for the bOTTR map to read
    pl.format(CRS84 + " POINT({} {})^^geo:wktLiteral", "long", "lat").alias("wkt"),
    # "scree;the-tarn" -> "book:scree;book:the-tarn"
    pl.col("stocks").str.split(";").list.eval(pl.lit("book:") + pl.element()).list.join(";"),
)

A join, twice. A trail segment's label and line come from the shops at its two ends, which is a join of segments.csv to the shops, once for each end:

ends = shops.select("shop", "name", "lat", "long")
segments = (
    read("segments.csv")
    .join(ends.rename({c: f"from_{c}" for c in ends.columns}), left_on="from", right_on="from_shop")
    .join(ends.rename({c: f"to_{c}" for c in ends.columns}), left_on="to", right_on="to_shop")
    .with_columns(
        pl.format("{} to {}", "from_name", "to_name").alias("label"),
        pl.format(CRS84 + " LINESTRING({} {}, {} {}, {} {})^^geo:wktLiteral",
                  "from_long", "from_lat", "via_long", "via_lat", "to_long", "to_lat").alias("wkt"),
        …
    )
)

A bOTTR map's H2 query cannot join (Chapter 8), and a TARQL query sees one row at a time; a data frame does it in a line.

Lists, from rows. A shop's stock is 95 rows in stock.csv; tt:ShopStock wants one row per shop with four lists. group_by and agg gather them, with typed values:

stock = (
    read("stock.csv")
    .with_columns(
        (pl.lit("stock:") + pl.col("shop") + "--" + pl.col("book")).alias("record"),
        (pl.lit("book:") + pl.col("book")).alias("book"),
        pl.col("copies").cast(pl.Int64),
        pl.col("shelf_price").cast(pl.Decimal(10, 2)),
        prefixed("shop", "shop").alias("shop"),
    )
    .group_by("shop", maintain_order=True)
    .agg(pl.col("record").alias("records"), pl.col("book").alias("books"),
         pl.col("copies"), pl.col("shelf_price").alias("prices"))
)
stock.write_parquet(OUT / "shop-stock.parquet")

Prices are decimals with two places, not floats: 9.99 is exactly 9.99, as money should be. The CSV copy joins each list with ;, because CSV cannot hold a list; the Parquet file holds real lists, and that is the file to prefer.

python python/prepare.py
data/prepared/bookshops.csv: 9418 bytes
data/prepared/segments.csv: 8143 bytes
data/prepared/shop-stock.csv: 6354 bytes
data/prepared/shop-stock.parquet: 5174 bytes

Mapping a data frame

import polars as pl
import tarka_rdf

shops = pl.read_csv("data/bookshops.csv", infer_schema=False)
query = tarka_rdf.Mapping.tarql_file("tarql/bookshops.rq")
triples = query.triplify(shops)

A Mapping is a compiled mapping: Mapping.tarql_file(path) or Mapping.tarql(text) for a TARQL query, Mapping.ottr(library, roots) for templates. triplify gives the triples as a frame of N-Triples terms, with the columns subject, predicate and object and no duplicates, so Polars can look at them before anything is written. python/frames.py prints Marginalia's first six:

bookshops: 33 rows, 588 triples
| subject            | predicate    | object            |
|--------------------|--------------|-------------------|
| bt:shop-marginalia | rdf:type     | bs:Bookshop       |
| bt:shop-marginalia | rdf:type     | geo:Feature       |
| bt:shop-marginalia | rdfs:label   | "Marginalia"@en   |
| bt:shop-marginalia | bs:locatedIn | bt:place-wigtown  |
| bt:shop-marginalia | bs:founded   | "1994"^^xsd:gYear |
| bt:shop-marginalia | bs:floorArea | "85"^^xsd:decimal |

(shortened for print: the frame holds full IRIs). write writes a file, or returns the text when no path is given:

query.write(shops, "out/bookshops-frame.ttl")              # Turtle
query.write(shops, "out/shops.nt", format="ntriples")
text = query.write(shops, format="nquads", graph="https://example.org/bookshop-trail/graph/shops")
query.run_csv("data/bookshops.csv", "out/bookshops.ttl")   # straight from CSV, as the command reads it

The 588 triples are the command's 588. A TARQL query binds every cell as a string, as it does for CSV, which is why the frame was read with infer_schema=False.

Parquet and list columns, from Python

stock = pl.read_parquet("data/prepared/shop-stock.parquet")
print(stock.schema)
templates = tarka_rdf.Mapping.ottr(["ottr"], ["tt:ShopStock"])
templates.write(stock, "out/shop-stock-frame.nt", format="ntriples")
Schema({'shop': String, 'records': List(String), 'books': List(String), 'copies': List(Int64), 'prices': List(Decimal(precision=10, scale=2))})

The list columns go to the templates' list parameters as they are; a text column for a list parameter needs its separator instead, Mapping.ottr(library, roots, lists={"records": ";"}), as an rdf:langString parameter needs its column's language, languages={"name": "en"} (Chapter 7). A frame whose columns are not named for the parameters is refused before any row is made, with the nearest column named; Polars' rename is the cure. And because the triples are a frame, a question can be asked before any store sees them: which shops hold the most stock records?

made = templates.triplify(stock)
busiest = (
    made.filter(pl.col("predicate") == "<https://example.org/bookshop-trail/schema#atShop>")
    .group_by("object")
    .len("records")
    .sort(["records", "object"], descending=[True, False])
    .head(5)
)
| object                | records |
|-----------------------|---------|
| bt:shop-ex-libris     | 6       |
| bt:shop-castle-steps  | 4       |
| bt:shop-colophon      | 4       |
| bt:shop-cotton-quarto | 4       |
| bt:shop-endpapers     | 4       |

How Polars types become cells

The same rules apply to a frame in Python and to a Parquet or Arrow file given to the command:

Polars type A cell holds
String, Categorical, Enum the string (empty or blank: no value)
integers, Decimal, Boolean the value: 42, 9.95, true
Float32, Float64 the shortest form that reads back as the value; INF, -INF, NaN
Date, Time 2024-03-05, 10:15:00
Datetime 2024-03-05T10:15:00; in UTC, ending in Z, when the column has a time zone
Duration PT90S
List a list, for an OTTR list parameter (a TARQL query cannot bind one)
null no value

A typed OTTR parameter reads a cell as its type, so an integer column feeding an xsd:integer parameter gives "42"^^xsd:integer, and a list keeps its items' types: a list of decimals gives 9.99, a list of dates 2025-03-08. (Before tarka 0.3.1, a list's items came through as Polars stores them, a decimal as its scaled integer.)

When a row cannot be made

A row with a cell its parameter cannot take is left out, the others are made, and a ValueError lists the rows once the rest is done:

ValueError: 1 row left out:
row 3: column shop: 'Border Print' is not an IRI

strict=False makes the triples it can and raises nothing:

templates.triplify(stock, strict=False)     # 30 triples from the two good rows

write and run_csv take strict too; with a path, the file is written before the error is raised, so the good rows are not lost.

The rest of the package

m.name, m.prefixes                    # the plan's name; the prefixes, for Turtle
m.shapes(base="https://example.org/bookshop-trail/shapes#")    # SHACL shapes, as Turtle
tarka_rdf.expand(["ottr"], ["instances/two-rivers.stottr"], "out/two-rivers.ttl")
tarka_rdf.bottr(["ottr"], ["bottr/bookshops.bottr.ttl"], "out/bookshops-bottr.ttl")

Each gives the same RDF as the command. The mapping runs without Python's global interpreter lock, on every core, so a large frame is mapped in parallel while other threads go on. Appendix B lists every function and argument.