Playbook: Knowledge Base Health¶

Objective¶

Maintain content quality, freshness, and discoverability across the knowledge base through regular audits and automated checks.

Quality audit script (scripts/audit_docs_quality.py)
Docs contract checker (scripts/check_docs_contract.py)
Catalog consistency checker (scripts/check_catalog_consistency.py)
Doc freshness checker (scripts/check_doc_freshness.py)
API pricing summary generator (scripts/update_api_pricing_capability_summary.py)
Model account policy validator (scripts/validate_model_account_pool.py)
Starred repo intake checker (scripts/check_starred_repo_intake.py)
Standards reference

Check	Frequency	Owner	How
Intake queue (`new-sources/`)	Daily	Jules (automated)	`daily-jules-maintenance.yml` opens a structured issue
Doc contract CI gate	Every PR	CI	`docs-quality-gates.yml` runs `check_docs_contract.py`
Catalog consistency CI gate	Every PR	CI	`catalog-quality-gates.yml` runs `check_catalog_consistency.py`
Generated pricing summary gate	PRs touching pricing tracker	CI	`generated-content-gates.yml` runs summary sync + freshness checks
API pricing maintenance	Weekly	CI	`api-pricing-maintenance.yml` refreshes capacity summaries and flags stale review metadata
External link health	Weekly + docs PRs	CI	`docs-link-health.yml` (Lychee) checks markdown links
Model-account routing policy gate	PRs touching policy file	CI	`model-account-policy-gates.yml` validates multi-account routing rules
Vikunja bidirectional sync	Disabled	Maintainer	Workflow intentionally removed from CI until the integration is reviewed and re-enabled safely
Full quality audit	Weekly (manual)	Maintainer	`python3 scripts/audit_docs_quality.py`
Starred repo drift check	Weekly (manual/local)	Maintainer	`python3 scripts/check_starred_repo_intake.py --ai-only --min-stars 5000`
Staleness review (docs >90 days old)	Monthly	Maintainer	See "Staleness check" below
Taxonomy alignment	Quarterly	Maintainer	Verify category dirs match `standards.md`

Run the audit script:
```
python3 scripts/audit_docs_quality.py
```
Review the output:
- Legacy-format docs: prioritise upgrading high-traffic pages first (tools you reference in playbooks or architecture docs).
- Missing metadata: add Last reviewed / Confidence / Sources blocks.
- Per-category breakdown: identify categories with the lowest compliance rate.
Fix the top 5 issues: focus on the docs that will fail CI the next time they are touched.
Update data/all_tools.json: ensure every page in mkdocs.yml nav has a matching entry.
Verify nav ↔ index consistency: each index.md in docs/tools/*/ should list all sibling tool pages.

Check starred-repo drift:

python3 scripts/check_starred_repo_intake.py --ai-only --min-stars 5000

Find docs not reviewed in 90+ days:

grep -rl "Last reviewed:" docs/ | xargs grep "Last reviewed:" | \
  awk -F': ' '{print $NF, $1}' | sort | head -20

For each stale doc, decide:
- Still accurate — update the Last reviewed date.
- Needs refresh — update content and bump the date.
- Obsolete — remove from mkdocs.yml, data/all_tools.json, and the category index.md.

Track these over time to measure knowledge base health:

Metric	Target	How to measure
Template compliance rate	>90%	`audit_docs_quality.py` → compliant / total
Legacy-format docs remaining	0	`audit_docs_quality.py` → legacy count
Docs with metadata	100%	`audit_docs_quality.py` → missing metadata count
Average doc age (days since last review)	<60 days	`grep "Last reviewed"` across all docs
Catalog consistency (nav ↔ JSON)	100%	`check_catalog_consistency.py` exit code
Intake queue backlog	0 `new` items	`grep "new" docs/new-sources/*.md`

Category index out of sync: a new tool doc is added to mkdocs.yml but not to its index.md. Fix: update the index file whenever adding a nav entry.
Orphaned JSON entries: a tool page is deleted but its all_tools.json entry remains. Fix: always update both when removing a page.
Duplicate pages: two pages document the same tool. Fix: merge into the canonical page and redirect/remove the duplicate.
Stale model references: docs reference old model names (e.g., "Claude 3.5 Sonnet" instead of "Claude Sonnet 4.6"). Fix: search-and-replace during staleness reviews.
Starred-repo drift: you star new GitHub repos but never stage them into docs/new-sources/. Fix: run check_starred_repo_intake.py locally and append any real candidates to today's daily log.