Skip to content

DataHerb catalog

DataHerb is a static data catalog: one YAML file points it at git repos and S3 buckets, and it builds a site where people can search datasets, preview and query them in the browser, and see which jobs are failing or stale. A gleanyard project can list itself there with no extra code. Add a dataherb block to gleanyard.yaml. The repo template already has one.

dataherb:
  id: berlin-family-events   # dataset and job id; defaults to the folder name
  expected_interval: P1D     # how often collect runs, as an ISO 8601 duration
  tags: [berlin, family, events]

Every key is optional: name and description default to the schema's, license to the one licence all exportable sources share, uri to the GitHub repo when running in Actions, owner, max_duration (when a running job counts as stuck), and the paths metadata (dataherb.json) and status_dir (_dataherb/status).

What gets written

gleanyard export writes dataherb.json, the dataset metadata (DataHerb dataset spec v2, a Frictionless Data Package underneath). It lists the export file with its row count, size and a schema built from schema.yaml: field types map to Table Schema types, id is the primary key, and _source carries provenance. Personal data never appears, because export never writes it. When no enabled source may be exported (for example all are internal_only), the metadata lists no files and says internal, so the catalog shows the dataset without offering a download. With export_csv set in gleanyard.yaml, the CSV copy of the export is listed too, so a DataHerb dataset can keep the CSV path its users already read.

gleanyard collect writes a job status (dataherb.status/v1) next to status.json: _dataherb/status/<id>/latest.json, plus one file per run in runs/. It reads:

DataHerb From gleanyard
run.status success when every source worked, partial when some failed, failed when all did
run.id, run.url, run.trigger the GitHub Actions run, when there is one
last_success carried over from the previous latest.json
checks one per enabled source: pass when ok, fail when degraded or failing, plus a warn per quiet-breakage warning (record_drop, fill_rate_drop, ...)
metrics sources, failing and degraded counts, records, rejections, pages, model calls, cost

DataHerb turns a failed check into a degraded job, and a missing successful run within expected_interval into a stale one, so the catalog shows the same health as the status hub. --from-cache runs write nothing, as with status.json. The hub can read the job status itself too, which is how it watches DataHerb datasets that are not gleanyard projects (see DataHerb job statuses).

Listing the repo in a catalog

The template's collect workflow commits dataherb.json and _dataherb/, so a DataHerb catalog reads both from git. In the catalog repo, add one entry file for the dataset and the job to the status sources in dataherb.config.yml (github is a store of type: git):

catalog/berlin-family-events.md
---
id: berlin-family-events
repo: kausalflow/berlin-family-events
---
dataherb.config.yml
status:
  sources:
    - store: github
      prefix: kausalflow/berlin-family-events@main:_dataherb/status
      jobs: [berlin-family-events]

A git store cannot list files, so the catalog shows the latest run; the runs/ history is read when the status folder is synced to S3 or served from a local folder.

To turn it off, delete the dataherb block; nothing else depends on it.