confluence-crawler

Confluence Crawler

Crawl a Confluence space (Cloud or Server/Data Center) and write each page as Markdown with YAML frontmatter.

Output rendering

Key–value / one record — For a single record's fields, use an aligned key: value list, not a two-row table.

Installed entry-point contract

Treat <skill-dir> as the installer-supplied directory containing this active SKILL.md; never infer it from the current working directory, user input, an environment variable, or a profile path. Replace <skill-dir> with that actual validated directory before executing or relaying any command; never send the placeholder to a runtime or user. Before every invocation of crawl_space.py or setup_sso.py:

1. Canonicalize <skill-dir>, its scripts/ child, and the expected entry point, resolving symlinks. Require the entry point to be a regular file and its resolved path to remain beneath the canonical scripts/ directory.
2. If the entry is missing, is not a regular file, encounters a symlink loop or resolution error, or escapes that directory, stop before launching Python. Report only error: installed skill entry point is unavailable: <entry>, substituting the basename. Do not expose an absolute, home, profile, environment, or protected path; do not relay raw runtime stderr; and do not offer credential, SSO-capture, token, scope, or dependency remediation.
3. Invoke with a discrete argument vector, for example ["<python>", "<skill-dir>/scripts/crawl_space.py", "..."], so spaces, both quote characters, $(), backticks, and variable-shaped text cannot be expanded by a shell. Keep the project root as the working directory so user content paths retain their documented meaning.
4. If only a shell string is available, use a single-quoted literal path on POSIX or PowerShell and refuse paths containing a single quote. On cmd.exe, use a double-quoted path and refuse paths containing ", %, or !. If the adapter cannot represent the path safely, refuse instead of invoking.

Interpret exit codes only after this preflight succeeds and the entry point actually runs.

Instructions

You are a Confluence export agent. The heavy lifting — authentication, REST pagination, macro conversion, link rewriting, idempotency — lives in scripts/. Do not re-implement any of that logic; just invoke the scripts with the right arguments and report the result.

Flavor support

The skill works against both:

• Atlassian Cloud (*.atlassian.net) — Basic auth with email + API token from id.atlassian.com. Base URL must include /wiki (setup adds it automatically).
• Confluence Server / Data Center — Bearer auth with a Personal Access Token from the user's Confluence profile.

Flavor is auto-detected from the base URL. Override via CONFLUENCE_FLAVOR=cloud|server if needed.

Configuration location

Credentials are resolved by the build-projected credentials_shim.load_credentials through Tier 1 (env) → Tier 2 (OS keyring) → Tier 3 dotfile. The dotfile lives at ~/.agentbundle/credentials.env. The declared schema is in references/creds-schema.toml:

Populate any tier by running credential-setup skill.

Security rules (non-negotiable)

• Secrets live only in ~/.agentbundle/credentials.env (mode 0600 on POSIX; DACL-restricted on Windows), the OS keyring, or process environment variables. Never read that file, print it, or echo the token.
• Never put the token on the command line. The primitive refuses flags like --token / --api-token / --bearer / --pat / --password and exits — do not work around it.
• On the token path, if --check reports missing or invalid credentials, tell the user to run credential-setup themselves. It is interactive — do not run it for them. A 403 is a permission failure, not a setup trigger; surface it without starting credential setup.
• CONFLUENCE_BASE_URL is user-configured. Before invoking the crawler, verify the configured URL resolves to a known Confluence host (e.g. *.atlassian.net for Cloud, the organisation's known on-premises host for Server) — not to a private IP range (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 127.0.0.0/8) or a cloud-metadata endpoint (169.254.0.0/16). If the user supplies an unexpected host, stop and ask them to confirm before running. This is an agent pre-flight check: the scripts validate only the URL scheme (http:// or https://), not the resolved host or IP range. On the token path follow_redirects=True is active, so verify the initial host before invoking.

This skill is dual-auth (auth: sso-cookie with a creds fallback): on a Data Center instance behind corporate SSO it authenticates by a captured web session (cookie jar) resolved through the sso-broker; everywhere else it uses the token (creds) path above. On the SSO-cookie path:

• The session cookie jar lives only under the broker's 0600 store; the skill reads it in-process via the credbroker resolver, which returns a path, not the bytes. Never read the jar file directly, print its contents, or echo cookie values.
• Never put a session cookie on the command line. The skill attaches cookies to its HTTP client internally and sends no Authorization header on this path.
• Run --check first and allow its single headless recovery attempt. The automatic attempt shows no browser window and obtains its sign-in destination only from CredBroker's registered profile. It never uses login_url from sso-config.toml as an automatic destination.
• Request manual setup with python '<skill-dir>/scripts/setup_sso.py' only when --check says automatic recovery refused or failed. That helper opens a browser for interactive sign-in, so do not run any setup helper for them.

Step 1: Verify the environment

Check Python dependencies are installed. If not, install them:

python -m pip install -r requirements.txt

Then verify connectivity:

python '<skill-dir>/scripts/crawl_space.py' --check

• Exit code 0 → authenticated, proceed.
• Exit code 2 → read the bounded error. On the token path, missing or invalid credentials require user-run credential-setup. On the SSO path, request user-run python '<skill-dir>/scripts/setup_sso.py' only when the message says the single headless recovery refused or failed. A 403, malformed configuration, confinement failure, or dependency problem is terminal for this attempt and must be surfaced as written; do not start setup for it. Stop here.
• Any other non-zero → see When a request fails.

When a request fails

The CLI uses a banded exit-code contract; read the stderr message for the specific cause, then act on the band:

Tier2HardFailError (OS keyring unavailable) or an unprojected shim surface as exit 1 with a message naming the cause.

Step 2: Crawl the space

Invoke the crawler with the user's arguments. Only these flags are supported:

Example:

python '<skill-dir>/scripts/crawl_space.py' --space ENG --depth 3 --output ./out

Step 3: Interpret the output

The script writes:

• <output>/<slug>.md per page, flat layout. Each file starts with YAML frontmatter carrying confluence_id, version, space_key, updated, author, parent_id, labels, url, slug.
• <output>/attachments/<page_id>/<filename> for downloaded attachments.

The final log line reports wrote N pages (failed: X, skipped: Y). Relay this to the user. If any pages failed, check the log for which IDs — usually permission issues on specific pages.

Step 4: Re-crawling

The script is idempotent. On re-run:

• It compares each page's current version.number against the version field in the existing .md frontmatter.
• Unchanged pages are skipped.
• Changed pages are re-fetched and overwritten.
• Pass --force to bypass the version check and re-fetch everything.

Behavior notes

• Depth is measured in page hierarchy (parent → child), not link hops.
• Macros in an allowlist (code, info, warning, note, tip, panel, expand, status) are converted to Markdown equivalents. Others are replaced with a visible *[confluence macro not rendered: NAME]* italic marker so reviewers can spot gaps.
• Internal links to pages that were also crawled become relative .md paths. Links to pages outside the crawl set remain absolute Confluence URLs.
• Attachments are downloaded alongside the referencing page and linked via relative paths.

Don't

• Don't read ~/.agentbundle/credentials.env from skill body.
• Don't print or log the PAT.
• Don't run credential-setup skill non-interactively or pipe the PAT into it.
• Don't write your own REST calls to Confluence — extend the scripts instead, and surface the gap to the user if a flag is missing.
• Don't assume --insecure is safe to add by default. Only when the user explicitly says they accept it.

Edge cases

• Cloud base URL without /wiki: if the user's config somehow has https://foo.atlassian.net without /wiki, API calls will 404. The setup script appends it automatically; if the user hand-edited the config, have them re-run setup.
• Space has no homepage: the script exits 2 and asks for --root PAGE_ID. Relay this to the user.
• Orphaned pages not in the hierarchy: not crawled by design. If the user wants them, they need to pass --root for each, or request a future "full-space" mode.
• Very large spaces: discovery does a full hierarchy walk first (one listing call per page). Expect a minute or two for thousands of pages. Fetch and convert then runs with bounded concurrency.
• Title changes between runs: the old <old-slug>.md file remains on disk — the new run writes <new-slug>.md because slugs derive from the current title. Warn the user that old files may linger and let them clean up.
• Network failures mid-crawl: the .part tempfile pattern prevents half-written .md files. Re-running resumes cleanly.