corpus-grounding

corpus-grounding

The reusable grounding layer from ADR-061: one pure-stdlib engine (BM25 retrieval + structured pre-filters + decision-rule evaluation) that any domain plugs into via a manifest — frontend design is its first consumer (design-intelligence). A corpus grounds (pre-action, constrains the option space); it never replaces mid-action reference (RAG), post-action validation (rules), or the method itself (a framework skill).

Engine provenance: ported from nextlevelbuilder/ui-ux-pro-max-skill @ b7e3af80f6e331f6fb456667b82b12cade7c9d35 (MIT, last checked 2026-06-07) — BM25 de-duplicated, slide-only paths stripped, every frontend-hardcoded axis moved into the manifest. Full license obligations: design-intelligence/ATTRIBUTION.md.

When to use

• A skill needs pre-action selection grounded in curated knowledge ("which layout pattern / threat class / index strategy applies here").
• You are authoring a new domain corpus — write a manifest + CSVs, validate with the grounding CLI (ground validate via ./scripts-run), never fork the engine.
• You need stack-scoped Do/Don't guidance (--stack react …).

Do NOT

• Do NOT use the corpus for mid-task fact lookup — that is reference (references/ docs / RAG).
• Do NOT use it for output validation — write a rule/linter.
• Do NOT build a corpus that fits in 5 lines of an always-on rule.
• Do NOT fork the engine per domain — plug in via a manifest.
• Do NOT merge a manifest without owner + refresh cadence (validator refuses it anyway).

Procedure: Consult a corpus

1. Locate the domain manifest (<domain-skill>/data/manifest.json).
2. Run search (one domain / stack axis) or ground (reasoning plan).
3. Read confidence + evidence_gap before trusting any row.
4. Propose grounded options; the human confirms (Tier-1 default).

Invocation (consumer runtime)

Scripts resolve all paths skill-relative (per docs/contracts/skill-bundled-assets.md) — they work from any cwd:

./scripts-run <skills-root>/corpus-grounding/scripts/ground search \
  --manifest <skills-root>/<domain-skill>/data/manifest.json \
  "fintech dashboard" [--domain style] [--stack react] \
  [--filter "Severity=HIGH"] [--max-results 3] [--json]

./scripts-run <skills-root>/corpus-grounding/scripts/ground ground \
  --manifest <skills-root>/<domain-skill>/data/manifest.json \
  "luxury e-commerce" [--context '{"data_heavy": true}'] [--persist DIR] \
  [--variance 1-10] [--motion 1-10] [--density 1-10]

./scripts-run <skills-root>/corpus-grounding/scripts/ground validate --manifest …

<skills-root> is wherever skills are deployed (~/.claude/skills/ for Claude Code installs; src/skills/ inside this repo).

The three ground dials are optional 1-10 preferences, and unset means unset — a dial you do not pass changes nothing, so the no-flag result is byte-identical to what it was before they existed. --variance biases which style is selected from the rows the corpus returned (never which rows come back), and when it changes the pick away from the grounding rule's own choice the run says so in its evidence gaps. --motion needs a gsap domain in the manifest and reports when a tier matches nothing. --density overrides the spacing scale. All three render in the text output and in a persisted MASTER.md, not only under --json.

Output format

Interface contract v1 — stability promise. Per ADR-061 §2 ("Opus condition"), this interface is versioned; domains may depend on it. Breaking any item below requires a major bump (manifest_version: 2), a migration note here, and updates to every consuming skill in the same PR. Additive fields are allowed anytime.

Stable v1 surface:

1. Modules + public names — bm25_search.{BM25, load_csv, apply_filters, search_rows, RETRIEVERS}, decision_engine.{detect_domain, search_domain, search_stack, evaluate_rules, ground, persist_grounding}, schema_validator.{load_manifest, validate_manifest, resolve_data_path, ManifestError, MANIFEST_VERSION, TIERS}.
2. CLI — the grounding CLI (ground {search|ground|validate} via ./scripts-run) with the flags shown above.
3. Result shape — search: {domain|stack, query, file, count, results, scores, filtered_from, confidence{label,score}, evidence_gap[]}; ground: {domain, query, category, rule, rules_evaluation{matched, unmatched}, selections{<domain>: {best, alternatives, confidence}}, confidence{label,score}, evidence_gap[]}.
4. Contract invariants — every output carries confidence and evidence_gap; retrievers are name-selected (bm25 default, structured, hybrid); no network, no subprocess, read-only except the opt-in --persist DIR.

Manifest contract (schema-agnostic plug-in)

Each domain ships data/manifest.json beside its CSVs — declaring its own axes; the engine never assumes the frontend schema:

{
  "manifest_version": 1,
  "domain": "frontend-design",
  "tier": "conditional-grounding",
  "data_dir": ".",
  "retriever": "bm25",
  "default_domain": "style",
  "domains": {
    "style": {
      "file": "styles.csv",
      "search_cols": ["Style Category", "Keywords"],
      "output_cols": ["Style Category", "Best For"],
      "max_results": 3,
      "filters": {}
    }
  },
  "detect": { "style": ["minimalism", "glassmorphism"] },
  "stacks": { "react": "stacks/react.csv" },
  "stack_cols": { "search_cols": ["…"], "output_cols": ["…"] },
  "reasoning": {
    "file": "ui-reasoning.csv",
    "category_domain": "product",
    "category_column": "Product Type",
    "match_column": "UI_Category",
    "rules_column": "Decision_Rules",
    "priority_column": "Style_Priority",
    "priority_domain": "style",
    "name_columns": { "style": "Style Category" },
    "plan": { "style": 3, "color": 2 },
    "rules_module": "rules.ts"
  },
  "owner": "package-maintainer",
  "refresh_cadence": "quarterly",
  "upstream": { "repo": "…", "sha": "…", "last_checked": "YYYY-MM-DD" }
}

• tier ∈ lookup-only | conditional-grounding | constraint-emission; reasoning is only legal above lookup-only.
• owner, refresh_cadence, upstream{repo,sha,last_checked} are required — provenance discipline per ADR-061 §6; an unowned corpus is not merged.
• rules_module (optional TypeScript escape hatch where JSON rules cap out — a module exposing evaluate(rules, query, context); a legacy .py name in a manifest is remapped to its .ts/.js twin) must live beside the manifest — absolute paths and .. are refused.
• Decision rules are surfaced as matched and unmatched — the full rule space stays auditable, never a hidden gate.

Procedure: Author a new domain corpus

1. Pass the qualification rubric (ADR-061 §5): decision-rule utility beats row count; grounding must happen before action; "fits in 5 lines → it's a rule, not a corpus".
2. Write data/manifest.json + CSVs in your domain skill; run the grounding CLI (ground validate via ./scripts-run).
3. Name an owner + refresh cadence in the manifest, pin the upstream SHA.
4. Cite this skill from your domain skill; consult via the CLI (Tier-1 consultation default — propose grounded options, human confirms).

Gotchas

• An empty result is a legitimate outcome — surface the evidence gap and proceed on priors; never silently widen filters to force a hit.
• The structured filters pre-filter matches case-insensitive substrings; an over-specific value silently filters everything out — check filtered_from vs count when results look thin.
• detect keyword routing falls back to default_domain; product-shaped queries route best ("fintech dashboard"), generic words land on the default.
• BM25 tokenizer drops tokens ≤2 chars — "UI", "a11y" style queries need longer companions.

Security constraints (Step 1.6 runtime-safety record)

Four scripts ship under scripts/: ground.ts (CLI), bm25_search.ts, decision_engine.ts, schema_validator.ts.

• What they may touch — corpus CSVs and the manifest under the manifest's own directory. resolve_data_path refuses absolute paths and .. escapes, so the manifest directory is the read boundary.
• What they must never do — read outside that directory, reach the network, or spawn a subprocess. Embeddings and remote retrievers are intentionally not implemented (ADR-061 §2), and that is a containment decision, not a missing feature.
• Default invocation — read-only. The single write surface is opt-in --persist DIR, which writes MASTER.md (and pages/<page>.md when --page is given) under DIR/design-system/<project-slug>/. Nothing else writes.
• Outbound — nothing. No network access at all.

One qualified exception: rules_module executes a manifest-adjacent TypeScript module via dynamic import(). That is the same trust domain as the skill shipping the manifest, and its containment is resolve_data_path.

See also

• design-intelligence — first consumer (frontend corpus + manifest).
• ADR-061 — architecture, qualification rubric, fork resolutions.
• docs/contracts/skill-bundled-assets.md — how bundled scripts/ + data/ reach consumer runtime.