Automation with CLI and CI/CD

An OKF bundle is a directory of Markdown files. That turns knowledge curation into ordinary software work: pull requests, line-level diffs, review, CI gates. Everything true for source code is true here. This page shows the real three-stage pipeline that turns arbitrary websites into OKF v0.2 bundles, plus the git workflow and a conformance gate for CI.

The pipeline in three stages

[ Website / Blog / Sitemap ]
        │  1. blog_scraper.py
        ▼
[ HTML gesammelt, bereinigt, zu Markdown ]
        │  2. transform_to_okf.py
        ▼
[ OKF-v0.2-Frontmatter: Typen, Tags, Provenance, Trust ]
        │  3. build_data_json.py
        ▼
[ okf.md + llms.txt + INDEX.md + okf_data.json ]

The prerequisite is Python 3.10 or newer and a handful of libraries:

python3 -m venv venv
source venv/bin/activate
pip install requests beautifulsoup4 html2text trafilatura lxml

Stage 1 – scrape (blog_scraper.py)

The script walks pagination, archives, sitemaps, and single links, extracting only an article's main text, without ads, sidebars, and footers. Trafilatura is the precise main extractor, html2text the fallback when Trafilatura comes back empty. Each article lands as .md under domain/category/.

# Variante A: von Hub-URLs mit Auto-Discovery interner Artikel
python3 blog_scraper.py --urls "https://example.com/blog/" --output-dir articles_output --threads 10

# Variante B: aus einer Linkliste
python3 blog_scraper.py --url-file my_urls.txt --output-dir articles_output --threads 15

Stage 2 – transform (transform_to_okf.py)

This stage brings every file onto the OKF v0.2 profile. It derives the type from title and URL (Guide, Tool, CaseStudy, Article), generates tags from the topic, pulls a RAG-ready description from the first real paragraph, and adds the trust layer: generated, verified, status, stale_after, sources.

python3 transform_to_okf.py

The per-file result is exactly the frontmatter from the frontmatter schema – machine-generated, but valid v0.2.

Stage 3 – compile (build_data_json.py)

The third stage compiles all concepts into okf_data.json: the data source for web search, the D3 knowledge graph, and the RAG assistant. Alongside it, okf.md is generated as the master index and llms.txt as a signpost for AI crawlers.

python3 build_data_json.py
cp okf_data.json public/

One detail stages 2 and 3 share: the aggregate files okf.md, INDEX.md, README.md are never read as concepts. They are navigation.

The git-native workflow

Put articles_output/ in a repository. Every change to a concept is a commit, every addition a pull request. Because resource, generated, verified, and the links are plain text, a diff shows exactly what changed in the knowledge, not just that something changed. The trust fields make visible whether a change came by hand or from the pipeline.

Attaching a bundle to a user (admin CLI)

For operations there is a server-side script that attaches a finished bundle straight to a user's private, encrypted storage, without charging their monthly limit. The user must have signed in once so their Google identifier is in the database.

node scripts/ingest.js <email> <pfad-zur-okf-datei>

The script encrypts the bundle under the recipient's per-user key and stores only ciphertext on S3 – the same mechanism every upload uses, described under encryption.

Checking conformance in CI

The most valuable automation step is a gate that checks a bundle before merge. Because base conformance has only three hard rules, standard tooling suffices. The rules are on the validator page: parseable YAML frontmatter, a non-empty type per concept, correctly handled reserved files.

A GitHub Actions workflow that checks on every pull request has this shape at its core (illustrative; wire the check to your rules):

name: okf-conformance
on: [pull_request]
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - name: Frontmatter und type prüfen
        run: |
          # Jede nicht-reservierte .md muss gültigen Frontmatter
          # mit nicht-leerem type haben. Fehler => Exit-Code != 0.
          python scripts/check_okf.py ./articles_output

The check script walks the .md files, skips okf.md, INDEX.md, README.md, parses the block between the first two --- lines as YAML, and checks type. Report soft points – missing description, missing generated, broken links – as warnings without breaking the build. The spec explicitly requires consumers not to reject on those grounds.

Publishing as an artifact

After the green gate, distribution is a copy operation: a tarball for download, okf_data.json plus llms.txt on a static server, or an upload into the OKF Knowledge Hub for library, graph, and RAG. The Hub upload today runs through the signed-in web interface or the admin script above, not a public write API.

Continue

The conformance rules are on the validator page, the target schema in the frontmatter schema. The prompt that lets a model take over stage 2 is under prompt templates.

---

Pipeline nach dem realen OKF_AUTOMATION_GUIDE und den Skripten blog_scraper.py, transform_to_okf.py, build_data_json.py, scripts/ingest.js dieses Projekts. CI-Beispiel illustrativ. / Pipeline per this project's real OKF_AUTOMATION_GUIDE and the scripts. CI example is illustrative.