DataHerb catalog¶
DataHerb is a static data catalog: one YAML file points it
at git repos and S3 buckets, and it builds a site where people can search datasets, preview and
query them in the browser, and see which jobs are failing or stale. A gleanyard project can
list itself there with no extra code. Add a dataherb block to gleanyard.yaml. The repo
template already has one.
dataherb:
id: berlin-family-events # dataset and job id; defaults to the folder name
expected_interval: P1D # how often collect runs, as an ISO 8601 duration
tags: [berlin, family, events]
Every key is optional: name and description default to the schema's, license to the
one licence all exportable sources share, uri to the GitHub repo when running in Actions,
owner, max_duration (when a running job counts as stuck), and the paths metadata
(dataherb.json) and status_dir (_dataherb/status).
What gets written¶
gleanyard export writes dataherb.json, the dataset metadata (DataHerb dataset spec
v2, a Frictionless Data Package underneath). It lists the export
file with its row count, size and a schema built from schema.yaml: field types map to
Table Schema types, id is the primary key, and _source carries provenance. Personal data
never appears, because export never writes it. When no enabled source may be exported (for
example all are internal_only), the metadata lists no files and says internal, so the
catalog shows the dataset without offering a download. With export_csv set in
gleanyard.yaml, the CSV copy of the export is listed too, so a DataHerb dataset can keep
the CSV path its users already read.
gleanyard collect writes a job status (dataherb.status/v1) next to status.json:
_dataherb/status/<id>/latest.json, plus one file per run in runs/. It reads:
| DataHerb | From gleanyard |
|---|---|
run.status |
success when every source worked, partial when some failed, failed when all did |
run.id, run.url, run.trigger |
the GitHub Actions run, when there is one |
last_success |
carried over from the previous latest.json |
checks |
one per enabled source: pass when ok, fail when degraded or failing, plus a warn per quiet-breakage warning (record_drop, fill_rate_drop, ...) |
metrics |
sources, failing and degraded counts, records, rejections, pages, model calls, cost |
DataHerb turns a failed check into a degraded job, and a missing successful run within
expected_interval into a stale one, so the catalog shows the same health as the
status hub. --from-cache runs write nothing, as with status.json. The hub can read the job status
itself too, which is how it watches DataHerb datasets that are not gleanyard projects (see
DataHerb job statuses).
Listing the repo in a catalog¶
The template's collect workflow commits dataherb.json and _dataherb/, so a DataHerb
catalog reads both from git. In the catalog repo, add one entry file for the dataset and the
job to the status sources in dataherb.config.yml (github is a store of type: git):
---
id: berlin-family-events
repo: kausalflow/berlin-family-events
---
status:
sources:
- store: github
prefix: kausalflow/berlin-family-events@main:_dataherb/status
jobs: [berlin-family-events]
A git store cannot list files, so the catalog shows the latest run; the runs/ history is
read when the status folder is synced to S3 or served from a local folder.
To turn it off, delete the dataherb block; nothing else depends on it.