tokf-filter

tokf Filter Authoring

You are an expert at writing tokf filter files. tokf is a config-driven CLI that compresses command output before it reaches an LLM context. Filters are TOML files that define how to process a command's output.

When the user asks you to create or modify a filter, follow this guide exactly. Produce valid, idiomatic TOML that matches the schema described below.

Section 1 — What a Filter File Is

A filter file is a TOML file that describes:

• Which command(s) it applies to (command)
• How to transform the raw output (steps, applied in a fixed order)
• What to emit on success vs. failure

Filters live in three places, searched in priority order:

1. .tokf/filters/ — project-local (repo-level overrides)
2. ~/.config/tokf/filters/ — user-level overrides
3. Built-in library (embedded in the tokf binary)

First match wins. Use tokf which "cargo test" to see which filter would activate for a given command.

Section 2 — Processing Order

Steps execute in this fixed order — do not rearrange them:

1. match_output — whole-output substring checks; if matched, short-circuits the entire pipeline and emits immediately
2. [[replace]] — per-line regex transforms applied to every line, in array order
3. strip_ansi / trim_lines — per-line cleanup (ANSI stripping, whitespace trimming)
4. skip / keep — line-level filtering (drop or retain lines by regex)
5. dedup / dedup_window — collapse duplicate consecutive lines
6. lua_script — Luau escape hatch; runs after dedup, before JSON/section/parse
7. [json] — JSON extraction via JSONPath; when configured, replaces section/parse/chunk
8. [[section]] OR [parse] — structured extraction (these are mutually exclusive; section is a state machine, parse is a declarative grouper). Skipped when [json] is configured.
9. [[chunk]] — block-based structured extraction with per-block aggregation, grouping, and tree output (runs on raw output, alongside sections). Skipped when [json] is configured.
10. Exit-code branch — [on_success] or [on_failure] depending on exit code
11. [fallback] — if neither on_success nor on_failure produced output
12. strip_empty_lines / collapse_empty_lines — post-processing cleanup on the final output

Within [on_success] and [on_failure], fields are processed as:

• head / tail → trim lines
• skip / extract → further filter
• aggregate → reduce collected sections
• output → final template render

Section 3 — Top-Level Fields Reference

Section 4 — Step Types

4.1 match_output — Whole-Output Short-Circuit

Check the entire raw output for a substring. If matched, emit a fixed string and stop — no further processing.

match_output = [
  { contains = "Everything up-to-date", output = "ok (up-to-date)" },
  { contains = "rejected", output = "✗ push rejected (try pulling first)" },
]

• contains: literal substring to search for (case-sensitive)
• output: string to emit if matched
• {line_containing} template variable: the first line that contains the substring

match_output = [
  { contains = "error", output = "Error on: {line_containing}" },
]

When to use: for well-known one-liner outcomes that make the rest of filtering irrelevant (e.g., "already up to date", "nothing to push", "authentication failed").

4.2 [[replace]] — Per-Line Regex Transforms

Applied to every line, in array order, before skip/keep. Use to reformat noisy lines.

[[replace]]
pattern = '^(\S+)\s+\S+\s+(\S+)\s+(\S+)'
output = "{1}: {2} → {3}"

[[replace]]
pattern = '^\s+Compiling (\S+) v(\S+)'
output = "compiling {1}@{2}"

• pattern: Rust regex (RE2 syntax, no lookaheads)
• output: template with {1}, {2}, … for capture groups; {0} is the full match
• If the pattern doesn't match a line, that line passes through unchanged
• Invalid patterns are silently skipped at runtime

When to use: when a line contains useful information but in a verbose format — reformat it rather than dropping it.

4.3 skip / keep — Line Filtering

skip drops lines matching any regex. keep retains only lines matching any regex. They compose:

skip = [
  "^\\s*Compiling ",
  "^\\s*Downloading ",
  "^\\s*$",
]

keep = ["^error", "^warning"]

• Both are arrays of regex strings
• Applied after [[replace]]
• skip is checked first, then keep
• A line must pass both: not skipped, and (if keep is non-empty) matching keep

When to use: skip for removing known noise patterns; keep for allow-listing (e.g., keep only lines that start with error or warning).

Also available inside [on_success] and [on_failure] for branch-level filtering.

4.4 dedup / dedup_window — Deduplication

dedup = true           # collapse consecutive identical lines
dedup_window = 10      # dedup within a 10-line sliding window

• dedup = true: removes consecutive duplicate lines (like uniq)
• dedup_window = N: deduplicates within a sliding window of N lines (catches near-consecutive repeats)
• They are independent; you can use both

When to use: for commands that emit repetitive progress lines (e.g., npm install printing the same package multiple times, spinner frames, repeated warnings).

4.5 lua_script — Luau Escape Hatch

For logic that pure TOML cannot express: numeric math, multi-line lookahead, conditional branching.

[lua_script]
lang = "luau"
source = '''
if exit_code == 0 then
    return "passed"
else
    local msg = output:match("Error: (.+)") or "unknown error"
    return "FAILED: " .. msg
end
'''

Or load the script from an external file:

[lua_script]
lang = "luau"
file = "scripts/my-filter.luau"

The file path resolves relative to the current working directory. Exactly one of source or file must be set.

Globals available:

• output (string): the full output after skip/keep/dedup
• exit_code (integer): the command's exit code
• args (table of strings): the arguments passed to the command

Return semantics:

• Return a string → replaces output, skips remaining TOML pipeline
• Return nil → fall through to [[section]] / [parse] / [on_success] / [on_failure]

Sandbox: io, os, and package are blocked. No filesystem or network access. Standard math/string/table libraries are available.

When to use: only when no TOML step can express the logic. Most filters do not need this. Consider it after exhausting match_output, skip/keep, [[replace]], [[section]], and [parse].

4.6 [json] — JSON Extraction via JSONPath

For commands that produce JSON output (e.g. kubectl get pods -o json, gh api, docker inspect). Extracts values using JSONPath (RFC 9535) queries and produces template variables and structured collections.

[json]

[[json.extract]]
path = "$.items[*]"
as = "pods"

[[json.extract.fields]]
field = "metadata.name"
as = "name"

[[json.extract.fields]]
field = "status.phase"
as = "phase"

[on_success]
output = "Pods ({pods_count}):\n{pods | each: \"  {name}: {phase}\" | join: \"\\n\"}"

[[json.extract]] fields:

[[json.extract.fields]] fields:

Result mapping:

• Single scalar → vars["as_name"] = string_value (no count, no chunk)
• Array → ChunkData::Flat collection + {as_name_count} variable
• Objects without fields → top-level scalars auto-flattened into chunk items
• Objects with fields → named fields extracted per item

Pipeline behavior: when [json] is configured, [[section]], [parse], and [[chunk]] are skipped. JSON replaces line-based structural processing. Extracted vars and chunks flow into [on_success]/[on_failure] template rendering.

Error handling: invalid JSON input → extraction skipped, pipeline falls back to raw output (templates are not rendered). Invalid JSONPath → rule silently skipped, other rules still run. Empty array with fields → emits {as_name_count} = "0".

When to use: when the command produces structured JSON output and you need to extract specific fields. Prefer this over [parse] + skip/keep for JSON-native commands.

4.7 [[section]] — State-Machine Section Collector

The most powerful step. Defines a state machine that collects lines into named variables as it scans top-to-bottom.

[[section]]
name = "failures"
enter = "^failures:$"      # regex: start collecting when this matches
exit = "^failures:$"       # regex: stop collecting when this matches (after start)
split_on = "^\\s*$"        # regex: split collected lines into blocks on blank lines
collect_as = "failure_blocks"

[[section]]
name = "summary"
match = "^test result:"    # regex: collect only lines matching this (no enter/exit)
collect_as = "summary_lines"

Fields:

Accessing collected variables in templates:

When to use: when the output has distinct sections with clear start/end markers — test failure blocks, error sections, file change groups.

4.8 [[chunk]] — Block-Based Structured Extraction

Chunks split raw output into repeating structural blocks (e.g., per-crate test suites in a Cargo workspace), extract structured data per-block, and produce named collections for template rendering. Like sections, chunks operate on the raw (unfiltered) command output — skip/keep patterns do not affect chunk processing.

[[chunk]]
split_on = "^\\s*Running "       # regex marking the start of each chunk
include_split_line = true         # include the splitting line in the chunk (default: true)
collect_as = "suites_detail"      # name for the structured collection
group_by = "crate_name"           # merge chunks sharing this field value
children_as = "children"          # preserve original items as nested collection

[chunk.extract]
pattern = 'unittests.+deps/([\w_-]+)-'  # extract a field from the header line
as = "crate_name"
carry_forward = true              # inherit value from previous chunk when pattern doesn't match

[[chunk.body_extract]]
pattern = 'Running\s+(.+?)\s+\('
as = "suite_name"

[[chunk.aggregate]]
pattern = '(\d+) passed'          # aggregates run within each chunk's lines
sum = "passed"

[[chunk.aggregate]]
pattern = '^test result:'
count_as = "suite_count"

Fields:

carry_forward (on extract or body_extract): when a chunk's pattern doesn't match, inherit the value from the most recent chunk that did. Useful when boundary markers (like Running unittests) identify a group, and subsequent chunks should inherit that identity.

Structured collections in templates: each item has named fields accessible in each pipes:

[on_success]
output = """\
{suites_detail | each: "  {crate_name}: {passed} passed ({suite_count} suites)" | join: "\\n"}"""

Tree output with children_as: groups preserve their child items for nested template rendering:

[on_success]
output = """\
{suites_detail | each: "  {crate_name}: {passed} passed\\n{children | each: \"    {suite_name}: {passed} passed\" | join: \"\\n\"}" | join: "\\n"}"""

When to use: when output contains repeating structural blocks with per-block data you want to aggregate and display. Common for workspace build tools (Cargo, Gradle, Nx) where output is organized by sub-project.

4.9 [parse] — Declarative Structured Parser

Alternative to [[section]] for commands with table-like output. Declaratively extracts a header field and groups remaining lines.

[parse]
branch = { line = 1, pattern = '## (\S+?)(?:\.\.\.(\S+))?(?:\s+\[(.+)\])?$', output = "{1}" }

[parse.group]
key = { pattern = '^(.{2}) ', output = "{1}" }
labels = { "M " = "modified", "??" = "untracked", "D " = "deleted" }

[output]
format = """
{branch}{tracking_info}
{group_counts}"""
group_counts_format = "  {label}: {count}"
empty = "clean — nothing to commit"

[parse] fields:

[parse.group] fields:

[output] fields (used with [parse]):

When to use: for commands like git status, docker ps, kubectl get — table-formatted output where you want to extract a header and count/group rows.

4.10 [on_success] / [on_failure] — Exit Code Branches

These branches run after all top-level steps. They have their own sub-fields:

[on_success]
output = "ok ✓ {2}"          # template; collected variables are available
head = 20                     # keep first N lines
tail = 10                     # keep last N lines
skip = ["^\\s*$"]            # additional line filtering
extract = { pattern = '(\S+)\s*->\s*(\S+)', output = "ok ✓ {2}" }

# Singular form (one rule):
aggregate = { from = "summary_lines", pattern = 'ok\. (\d+) passed', sum = "passed", count_as = "suites" }

# Plural form (multiple rules):
# [[on_success.aggregates]]
# from = "summary_lines"
# pattern = 'ok\. (\d+) passed'
# sum = "passed"
# count_as = "suites"
#
# [[on_success.aggregates]]
# from = "summary_lines"
# pattern = '(\d+) failed'
# sum = "failed"

[on_failure]
tail = 10
output = "FAILED: {summary_lines | join: \"\\n\"}"

Branch sub-fields:

aggregate / aggregates fields:

Both singular aggregate and plural aggregates can be used together — they are merged at runtime.

When to use: Always. Every filter should have at least one of [on_success] or [on_failure]. Use [on_success] to produce a clean summary. Use [on_failure] to show enough context to diagnose the issue.

4.11 [fallback] — Last Resort

Emits output when neither [on_success] nor [on_failure] produced anything.

[fallback]
tail = 5

When to use: as a safety net when you have complex branching logic. Ensures tokf never silently swallows output.

4.12 [[variant]] — Context-Aware Filter Delegation

Some commands are wrappers around different underlying tools (e.g. npm test may run Jest, Vitest, or Mocha). A parent filter can declare [[variant]] entries that delegate to specialized child filters based on project context.

command = ["npm test", "pnpm test", "yarn test"]

strip_ansi = true
skip = ["^> ", "^\\s*npm (warn|notice|WARN|verbose|info|timing|error|ERR)"]

[on_success]
output = "{output}"

[on_failure]
tail = 20

[[variant]]
name = "vitest"
detect.files = ["vitest.config.ts", "vitest.config.js", "vitest.config.mts"]
filter = "npm/test-vitest"

[[variant]]
name = "jest"
detect.files = ["jest.config.js", "jest.config.ts", "jest.config.json"]
filter = "npm/test-jest"

Fields:

Two-phase detection:

1. File detection (before execution) — checks if any listed config files exist in the current directory. First match wins.
2. Output pattern (after execution) — regex-matches the command output. Used as a fallback when no file was detected.

At least one of detect.files or detect.output_pattern must be set.

Behavior:

• When a variant matches, the child filter replaces the parent entirely — no field inheritance or merging
• When no variant matches, the parent filter's own fields (skip, on_success, etc.) apply as the fallback
• The filter field references another filter by its discovery name (e.g. "npm/test-vitest" maps to filters/npm/test-vitest.toml)

TOML ordering: [[variant]] entries must appear after all top-level fields (skip, [on_success], etc.) because TOML array-of-tables sections capture subsequent keys.

When to use: when a single command pattern maps to different underlying tools that produce fundamentally different output formats. Create a parent filter with a generic fallback, then create specialized child filters for each tool.

Section 5 — Template Pipes

Output templates support pipe chains: {var | pipe | pipe: "arg"}.

Examples:

Filter a multi-line output variable to only error lines:

[on_failure]
output = "{output | lines | keep: \"^error\" | join: \"\\n\"}"

For each collected block, show only > (pointer) and E (assertion) lines:

[on_failure]
output = "{failure_blocks | each: \"{value | lines | keep: \\\"^[>E] \\\"}\" | join: \"\\n\"}"

Truncate long lines and number them:

[on_failure]
output = "{summary_lines | each: \"{index}. {value | truncate: 120}\" | join: \"\\n\"}"

Section 6 — Naming & Placement Conventions

File naming:

• filters/<tool>/<subcommand>.toml for two-word commands: filters/git/push.toml for git push
• filters/<tool>.toml for single-word commands: filters/pytest.toml for pytest
• For wildcards: filters/npm/run.toml with command = "npm run *" in the TOML
• Lowercase filenames only, no spaces

Placement:

When creating a filter for a user's project, default to .tokf/filters/ unless they specify otherwise.

Command field:

• Exact match: command = "git push" matches git push and git push origin main
• Wildcard: command = "npm run *" matches npm run dev, npm run build, etc.
• Array: command = ["cargo test", "cargo t"] matches either form

Section 7 — Workflow for Creating a New Filter

Follow these steps when asked to create a filter:

Step 1: Understand the command's output

Ask the user to provide (or capture) example output from the command. If they don't have it, generate a plausible example based on the tool's known output format. Look for:

• What's signal (errors, results, summaries)
• What's noise (progress bars, compilation lines, download progress, blank lines)
• What patterns mark sections (e.g., "failures:", "test result:")

Step 2: Choose the right complexity level

Start at the lowest level that handles the use case. Don't reach for [[section]] when skip + extract suffices.

Step 3: Draft the filter

1. Set command to match the command pattern
2. Add match_output for well-known short-circuit cases (empty output, auth failure, "already done")
3. Add skip to drop noise lines (progress, compile output, blank lines)
4. Add [[replace]] to reformat noisy-but-useful lines
5. Add [[section]] or [parse] if you need structured extraction
6. Write [on_success] with the desired output format
7. Write [on_failure] with enough context to diagnose (tail = 20 is a safe default)
8. Add [fallback] with tail = 5 as a safety net for complex filters

Step 4: Write test cases and verify

Create a <stem>_test/ directory adjacent to the filter TOML and add at least one test case per meaningful outcome (success, failure, edge cases):

filters/mytool/
  mysubcmd.toml       ← filter config
  mysubcmd_test/      ← test suite
    success.toml
    failure.toml

Each test case is a TOML file:

name = "success shows one clean line"
fixture = "tests/fixtures/mytool_success.txt"  # path relative to this file, then CWD
exit_code = 0

[[expect]]
equals = "ok ✓"

[[expect]]
not_contains = "noise"

Or with an inline fixture (no file needed):

name = "known error message"
inline = "Error: connection refused\n"
exit_code = 1

[[expect]]
contains = "connection refused"

Run the suite:

tokf verify mytool/mysubcmd   # run one suite
tokf verify                   # run all suites

For quick one-off testing without creating test files:

tokf apply filters/mytool/mysubcmd.toml tests/fixtures/mytool_output.txt --exit-code 0

Step 5: Place and name the file correctly

• Two-word command: .tokf/filters/mytool/mysubcmd.toml
• Single-word command: .tokf/filters/mytool.toml
• Wildcard command: .tokf/filters/mytool/run.toml with command = "mytool run *"

Section 8 — Three Annotated Examples

Example 1: git push (Level 1 — match_output + extract)

Goal: 15 lines of push noise → "ok ✓ main" (or failure message).

# filters/git/push.toml — Level 1
# Raw output: 15 lines of object counting, compression, "remote:" lines
# Filtered (success): "ok ✓ main"
# Filtered (up-to-date): "ok (up-to-date)"
# Filtered (rejected): "✗ push rejected (try pulling first)"

command = "git push"

# Check full output for well-known outcomes before any processing
match_output = [
  { contains = "Everything up-to-date", output = "ok (up-to-date)" },
  { contains = "rejected", output = "✗ push rejected (try pulling first)" },
]

[on_success]
# Drop all the noise lines
skip = [
  "^Enumerating objects:",
  "^Counting objects:",
  "^Delta compression",
  "^Compressing objects:",
  "^Writing objects:",
  "^Total \\d+",
  "^remote:",
  "^To ",
]
# Extract the branch name from the ref update line: "abc1234..def5678  main -> main"
extract = { pattern = '(\S+)\s*->\s*(\S+)', output = "ok ✓ {2}" }

[on_failure]
tail = 10

Key decisions:

• match_output handles the two most common "instant" outcomes
• extract captures the branch name from the ref update line
• tail = 10 on failure gives enough context without overwhelming

Example 2: git status (Level 2 — parse + group)

Goal: 30+ lines of verbose status → branch name + grouped file counts.

# filters/git/status.toml — Level 2
# Raw output: 30+ lines with hints, file paths, status codes
# Filtered: "main [ahead 2]\n  modified: 3\n  untracked: 2"

command = "git status"

# Override: use porcelain format for reliable machine parsing
run = "git status --porcelain -b"

match_output = [
  { contains = "not a git repository", output = "Not a git repository" },
]

[parse]
# First line: "## main...origin/main [ahead 2]"
# Extract: branch name, upstream, ahead/behind info
branch = { line = 1, pattern = '## (\S+?)(?:\.\.\.(\S+))?(?:\s+\[(.+)\])?$', output = "{1}" }

[parse.group]
# Group remaining lines by their two-character status code
key = { pattern = '^(.{2}) ', output = "{1}" }
labels = {
  "M " = "modified",
  " M" = "modified (unstaged)",
  "MM" = "modified (staged+unstaged)",
  "A " = "added",
  "??" = "untracked",
  "D " = "deleted",
  " D" = "deleted (unstaged)",
  "R " = "renamed",
  "UU" = "conflict",
  "AM" = "added+modified"
}

[output]
format = """
{branch}{tracking_info}
{group_counts}"""
group_counts_format = "  {label}: {count}"
empty = "clean — nothing to commit"

Key decisions:

• run overrides to porcelain format — machine-readable is easier to parse
• [parse] extracts the branch header line declaratively
• [parse.group] groups by status code without needing [[section]]
• [output] uses built-in {group_counts} variable populated by the parser

Example 3: cargo test (Level 4 — section + chunk + aggregates + tree)

Goal: 200+ lines with compile noise, per-test "ok" lines, failure blocks → per-crate tree summary on pass, structured failure report on fail.

# filters/cargo/test.toml — Level 4
# Raw output: 200+ lines
# Filtered (pass): "✓ cargo test: 1279 passed, 0 failed, 119 ignored (42 suites)"
#                   with per-crate tree breakdown showing individual test suites
# Filtered (fail): failure details + summary

command = "cargo test"
strip_ansi = true

# Drop all the noise
skip = [
  "^\\s*Compiling ",
  "^\\s*Downloading ",
  "^\\s*Downloaded ",
  "^\\s*Finished ",
  "^\\s*Locking ",
  "^running \\d+ tests?$",
  "^test .+ \\.\\.\\. ok$",   # individual passing tests
  "^\\s*$",
  "^\\s*Doc-tests ",
]

# State machine: collect the "failures:" section into blocks split by blank lines
[[section]]
name = "failures"
enter = "^failures:$"
exit = "^failures:$"
split_on = "^\\s*$"
collect_as = "failure_blocks"

# Collect "test result: ok/FAILED" summary lines (one per test suite)
[[section]]
name = "summary"
match = "^test result:"
collect_as = "summary_lines"

# Chunk processing: per-crate breakdown from "Running" headers.
# "unittests" lines define crate boundaries; integration test suites
# inherit the crate name via carry_forward.
[[chunk]]
split_on = "^\\s*Running "
include_split_line = true
collect_as = "suites_detail"
group_by = "crate_name"
children_as = "children"

[chunk.extract]
pattern = 'unittests.+deps/([\w_-]+)-'
as = "crate_name"
carry_forward = true

[[chunk.body_extract]]
pattern = 'Running\s+(.+?)\s+\('
as = "suite_name"

[[chunk.aggregate]]
pattern = '(\d+) passed'
sum = "passed"

[[chunk.aggregate]]
pattern = '(\d+) failed'
sum = "failed"

[[chunk.aggregate]]
pattern = '(\d+) ignored'
sum = "ignored"

[[chunk.aggregate]]
pattern = '^test result:'
count_as = "suite_count"

# Success: aggregate summaries + per-crate tree breakdown
[on_success]
output = "✓ cargo test: {passed} passed, {failed} failed, {ignored} ignored ({suites} suites)\n{suites_detail | each: \"  {crate_name}: {passed} passed ({suite_count} suites)\\n{children | each: \\\"    {suite_name}: {passed} passed\\\" | join: \\\"\\\\n\\\"}\" | join: \"\\n\"}"

[[on_success.aggregates]]
from = "summary_lines"
pattern = 'ok\. (\d+) passed'
sum = "passed"
count_as = "suites"

[[on_success.aggregates]]
from = "summary_lines"
pattern = '(\d+) failed'
sum = "failed"

[[on_success.aggregates]]
from = "summary_lines"
pattern = '(\d+) ignored'
sum = "ignored"

# Failure: show failure details + summary
[on_failure]
output = "✗ cargo test: {passed} passed, {failed} failed ({suites} suites)\n\nFAILURES ({failure_blocks.count}):\n{failure_blocks | each: \"\\n── {index}. ──\\n{value}\" | join: \"\\n\"}\n\n{summary_lines | join: \"\\n\"}"

[[on_failure.aggregates]]
from = "summary_lines"
pattern = '(\d+) passed'
sum = "passed"
count_as = "suites"

[[on_failure.aggregates]]
from = "summary_lines"
pattern = '(\d+) failed'
sum = "failed"

[fallback]
tail = 5

Key decisions:

• skip removes all per-test "ok" lines — only failures and summaries remain
• [[section]] collectors handle failure blocks and summary lines
• [[chunk]] splits on Running headers, extracts crate names from unittests lines
• carry_forward = true makes integration test suites inherit the crate name from the preceding unit test suite
• children_as = "children" preserves per-suite detail within each crate group
• [[on_success.aggregates]] (plural) sums passed/failed/ignored across all suite summary lines
• Nested each pipes produce tree output: crate → suites
• [fallback] catches edge cases (compile errors with no test output)

Section 9 — Writing Test Cases

Every filter in the standard library has a <stem>_test/ directory with declarative test cases. When writing or modifying a filter, write test cases alongside it.

Test case format

name = "success output is a single clean line"   # required, human-readable
fixture = "tests/fixtures/cargo_build_success.txt"  # path to raw output file
# inline = "some raw output\nline two"            # alternative: inline fixture
exit_code = 0                                    # optional, default 0
args = []                                        # optional, forwarded to filter

[[expect]]
equals = "ok ✓"          # exact match

[[expect]]
not_contains = "Compiling"  # noise must be gone

Assertion types

Every [[expect]] entry checks one assertion. A test case with multiple [[expect]] entries must pass all of them. A test case with no [[expect]] entries is an error.

What to test

For every filter, write at least:

• Success case: the happy path produces the expected one-liner or summary
• Failure case: a failing exit code produces enough context to diagnose
• Edge cases: cover each match_output branch (e.g., "up-to-date", "rejected")

Directory convention

filters/
  git/
    push.toml          ← filter config
    push_test/         ← test suite (identified by _test suffix)
      success.toml
      up_to_date.toml
      rejected.toml
      failure.toml

The _test suffix makes suite directories immediately identifiable in file listings and distinguishes them from filter category directories.

Section 10 — Common Mistakes to Avoid

1. Don't use keep when skip is enough. keep is an allow-list — it drops everything that doesn't match. Use it only when you want to radically filter to a specific type of line.

2. Escape backslashes in TOML strings. In regular strings, \\d means literal \d in the regex. In TOML raw strings ('...'), backslashes are literal. Use raw strings for complex patterns.

3. match_output is a short-circuit. If it matches, nothing else runs. Don't put it at the end expecting it to be a fallback — it runs first.

4. [[section]], [parse], and [json] are mutually exclusive in practice. [json] replaces [[section]]/[parse]/[[chunk]] — when [json] is configured, those line-based steps are skipped. Use [json] for JSON output, [[section]]/[parse]/[[chunk]] for line-based output.

5. {output} in branch templates is the filtered output text (after skip/keep/replace/dedup), not the raw command output.

6. Pipe chains need careful quoting. When nesting templates inside each:, escape inner quotes: {each: "{value | lines | keep: \\\"^error\\\"}"}.

7. Don't skip the [fallback]. Complex filters with [[section]] can produce empty output if sections don't match. Always add [fallback] tail = 5 as a safety net.

8. Test with realistic fixture data. A filter that works on a trimmed example may miss edge cases. Use real command output saved to a .txt fixture file.

Section 11 — Generic Commands (When a Filter Isn't Needed)

Before writing a full filter, consider whether a generic command would be sufficient. tokf provides three built-in subcommands that work on any command without a TOML filter:

When to use generic commands vs. writing a filter

• Use generic commands when the tool's output follows common patterns (error lines, test results, repetitive logs) and you don't need precise control over the output format.
• Write a filter when you need structured extraction, specific sections, custom templates, or the generic output isn't useful enough.

Routing generic commands through rewrites

To make generic commands trigger automatically via the hook, add rewrite rules to .tokf/rewrites.toml:

# Build tools without dedicated filters
[[rewrite]]
match = "^mix compile"
replace = "tokf err {0}"

# Test runners without dedicated filters
[[rewrite]]
match = "^mix test"
replace = "tokf test {0}"

# Long output without dedicated filters
[[rewrite]]
match = "^terraform plan"
replace = "tokf summary {0}"

Important: Only add rewrite rules for commands that don't already have a filter. Check with tokf which "<command>" first. Commands with dedicated filters produce better output through tokf run.