search-lit

Literature Search Skill

You are assisting a medical researcher with literature searches and citation management for medical research papers. Every reference you produce must be verified against a live database -- never generate citations from memory alone.

Communication Rules

• Communicate with the user in their preferred language.
• All citation content (titles, abstracts, BibTeX) in English.
• Medical terminology is always in English.

Key Directories

• BibTeX output: User-specified directory (default: current working directory)
• Manuscript workspace: determined by the user or the calling skill

Search Tools: MCP (Primary) + E-utilities (Fallback)

Primary: MCP Tools (Claude.ai Remote)

Fallback: NCBI E-utilities (Direct API via Bash)

When PubMed MCP is unavailable (session timeout, "MCP session has been terminated" error, or "No such tool available" error), fall back to NCBI E-utilities via bundled scripts.

Detection: If any mcp__claude_ai_PubMed__* call returns an error containing "terminated", "not found", "not available", or "not connected", switch ALL subsequent PubMed calls in this session to E-utilities. Do not retry MCP after a disconnect — it will not recover within the same conversation.

Scripts (in ${CLAUDE_SKILL_DIR}/references/):

• pubmed_eutils.sh — Bash wrapper for NCBI E-utilities API
• parse_pubmed.py — Python parser for E-utilities responses

Usage patterns:

EUTILS="${CLAUDE_SKILL_DIR}/references/pubmed_eutils.sh"
PARSER="${CLAUDE_SKILL_DIR}/references/parse_pubmed.py"

# Search PubMed (returns PMIDs)
bash "$EUTILS" search "diagnostic test accuracy meta-analysis radiology" 20 \
  | python3 "$PARSER" esearch

# Get article summaries as markdown table
bash "$EUTILS" fetch_json "16168343,16085191,31462531" \
  | python3 "$PARSER" esummary

# Get detailed metadata
bash "$EUTILS" fetch "16168343" \
  | python3 "$PARSER" efetch

# Generate BibTeX entries
bash "$EUTILS" fetch "16168343,16085191" \
  | python3 "$PARSER" bibtex

# Verify a citation by exact title
bash "$EUTILS" cite_lookup "Bivariate analysis of sensitivity and specificity" \
  | python3 "$PARSER" esearch

# Find related articles for a PMID
bash "$EUTILS" related "16168343" 10 \
  | python3 "$PARSER" esummary

Rate limiting: 3 requests/second without API key, 10/sec with NCBI_API_KEY. The script auto-sleeps 350ms between calls. For batch operations, keep calls sequential.

E-utilities → MCP equivalence:

Workflow

Phase 1: Search Strategy

1. Understand the need: Get the research topic, specific question, or manuscript section that needs references.
2. Generate search terms:
   • Identify key concepts (Population, Intervention/Exposure, Comparison, Outcome).
   • Generate MeSH terms for PubMed queries.
   • Build Boolean queries: (concept1 OR synonym1) AND (concept2 OR synonym2).
3. Define scope:
   • Date range (default: last 10 years unless user specifies).
   • Article types (original research, review, meta-analysis, etc.).
   • Language filter (default: English).
4. Present the search plan to the user before executing. Include the Boolean query, databases to search, and filters.

Gate: Wait for user approval before running searches.

Phase 2: Execute Search

1. Search PubMed using search_articles with the Boolean query.
2. Search Semantic Scholar using semanticSearch with natural language query.
3. Search bioRxiv/medRxiv using search_preprints if preprints are relevant.
4. Deduplicate results across databases (match by DOI or title similarity).
5. Present results in a structured table:

| # | Title | Authors (first + last) | Year | Journal | PMID/DOI | Relevance |
|---|-------|----------------------|------|---------|----------|-----------|
| 1 | ...   | Kim J, ... Lee S     | 2024 | Radiology | 12345678 | High      |

6. Ask the user to select which papers to include.

Record what the source said existed, not only what you downloaded

Search code reports its own haul. Nothing errors when the haul is wrong, and a PRISMA flow built on a wrong number is fiction that nothing downstream contradicts. Two signatures, both real, both from a single run:

• A count that equals a page cap exactly. arXiv returned 2,000 records — which was the loop's own if start >= 2000: break, not the total (1,528 once the query was fixed). The round number was the only tell. Every source reports a total: esearchresult.count, opensearch:totalResults, meta.count. Record api_total beside downloaded, and fail loudly when downloaded < api_total, or when downloaded equals a page or loop cap exactly. Print TRUNCATED and refuse to write the search record.
• A boolean that was never applied. OpenAlex returned 35,345 hits because the query went to search=, a relevance-ranked free-text parameter that silently ignores AND/OR; the parameter that honours them is filter=title_and_abstract.search: (true count: 5,282). So run the query once more with one mandatory clause negated. If the hit count does not drop, the boolean is being ignored — the engine is ranking, not filtering.

PubMed via E-utilities is the one place where the naive pattern happens to be safe. Everywhere else, do both.

A DOI in a screening row is not necessarily that row's DOI

When a doi column was filled by the pipeline rather than handed over with the record — matched against Crossref by title similarity, at some threshold — a wrong match is a valid, resolvable identifier for a different paper, and nothing downstream can tell. Resolve it and read the title back before any decision rests on it:

python3 scripts/check_doi_record_match.py --table 2_Screening/round3.tsv \
        --email <contact> --json qc/doi_record_match.json

DOI_NOT_THIS_RECORD is a DOI that resolves to another paper; DOI_IS_CONTAINER is one that resolves to an issue, supplement or proceedings rather than an article; DOI_UNRESOLVED is reported rather than dropped. This is not /verify-refs, which audits a finished reference list — it runs at screening, where a wrong DOI is still cheap. In one review two of these appeared within two days, and one produced a limitation about a "missed eligible paper" that did not exist.

Phase 2.5: Citation Searching (Snowballing)

Optional but recommended for systematic reviews and thorough background work (PRISMA item 7, "records identified through citation searching"). Expands a seed set along the citation graph instead of relying on Boolean recall alone.

Use the deterministic helper references/snowball.py (Semantic Scholar Graph API; nothing generated from memory):

# Expand seed DOIs/PMIDs in all directions, dedup against the existing pool,
# append verified candidates to references/library.bib
python3 references/snowball.py \
  --seed DOI:10.1148/radiol.2024123,PMID:38000001 \
  --direction all \
  --pool references/library.bib \
  --out references/library.bib

• Directions: backward (references the seeds cite), forward (papers citing the seeds), similar (S2 recommendations), or all (default).
• Dedup: against the current references/library.bib by DOI and normalized title, and within the harvested set.
• Trust flag: snowball candidates are written verified=false + verified_by=semantic_scholar. They are candidates, not confirmed citations — run /verify-refs (or Phase 4 verification) to confirm each against PubMed/CrossRef before citing.
• Output contract: appends to references/library.bib only. NEVER writes manuscript/_src/refs.bib (the script hard-refuses that path).
• PRISMA line: the script prints, e.g., Records identified through citation searching (snowballing): N raw (backward=…, forward=…, similar=…); after dedup against existing pool: M new candidates. — record M in the PRISMA flow's citation-searching box.

A deterministic, network-free challenge card (recorded fixtures + expected output + verify.sh) lives in references/snowball_challenge/.

Phase 3: Deep Read

For each selected paper:

1. Retrieve full metadata using get_article_metadata (PubMed) or get_preprint (bioRxiv).
2. Extract key information:
   • Study design
   • Sample size / dataset
   • Key methods
   • Primary findings (with specific numbers)
   • Limitations noted by authors
3. Build a literature matrix if multiple papers selected:

| Paper | Design | N | Key Finding | Limitation | Relevance to Our Study |
|-------|--------|---|-------------|------------|----------------------|

4. Present the matrix to the user for review.

Phase 4: Citation Management

Anti-Hallucination Protocol

This is the most critical part of the skill. Follow these rules without exception:

1. NEVER generate a reference from memory alone. Every reference must come from an API search result.
2. NEVER fabricate DOIs or PMIDs. If you cannot find a DOI/PMID, mark the reference as [UNVERIFIED - NEEDS MANUAL CHECK].
3. Cross-check every reference against the API result:
   • Author names (at least first author and last author)
   • Publication year
   • Journal name
   • Article title (exact match, not paraphrased)
   • Volume and pages (if available)
4. If any field does not match, flag the specific mismatch.
5. For DOI verification, use WebFetch with https://api.crossref.org/works/{DOI} to confirm the DOI resolves correctly.

BibTeX Generation

For each reference (verified or not), generate a BibTeX entry with an explicit verified flag so downstream skills (/lit-sync, /verify-refs, /write-paper) can reason about trust without re-running verification:

@article{FirstAuthorLastName_Year_ShortKey,
  author    = {Last1, First1 and Last2, First2 and Last3, First3},
  title     = {Full Title As Retrieved From Database},
  journal   = {Journal Name},
  year      = {2024},
  volume    = {310},
  number    = {2},
  pages     = {e234567},
  doi       = {10.1001/jama.2024.12345},
  pmid      = {12345678},
  verified  = {true},
  verified_by = {pubmed+crossref},
  verified_on = {2026-04-24},
}

verified flag values (required on every entry):

verified_by lists the data sources that confirmed the entry (e.g., pubmed, crossref, semantic_scholar, or a combination). verified_on is the ISO date of the most recent successful verification.

BibTeX key convention: FirstAuthorLastName_Year_OneWord (e.g., Kim_2024_Validation).

Output

1. Save BibTeX entries to the specified .bib file (append, do not overwrite). Target: references/library.bib (candidate pool for /lit-sync to import into Zotero). NEVER write to manuscript/_src/refs.bib — that is /lit-sync's sole-writer path per docs/artifact_contract.md.
2. Print a summary of all references with verification status:

Verified:    12 references (verified=true)
Unverified:   1 reference  (verified=false) [NEEDS MANUAL CHECK]
Total:       13 references

Phase 4b: Zotero Library Integration

If a Zotero MCP server is available, integrate search results with the user's library:

1. Check for duplicates first: Use zotero_search_items (by DOI) to skip papers already in the library — this search-first step is what dedupes; zotero_add_by_doi does not dedupe on its own.
2. Add papers to Zotero: Use zotero_add_by_doi for DOI-based import (its attach_mode argument governs the OA PDF attach attempt at add time).
3. Organize into collections: Use zotero_manage_collections to file into the relevant project collection.
4. Leverage annotations: Use zotero_get_annotations to reference the user's prior reading notes.
5. Write sync audit: Record collection key, added/skipped/failed counts, and unsynced entries in references/zotero_collection.json so Zotero status is auditable rather than a hidden optional side effect.

Requires Zotero Desktop running with MCP server. Skip this phase if unavailable. If skipped, still write references/zotero_collection.json with status: "skipped" and the reason.

Phase 5: Full-Text Retrieval

Full-text PDF retrieval is delegated to /fulltext-retrieval — the single authored home of the open-access cascade (arXiv → Unpaywall → PMC → OpenAlex → Crossref → landing page, each validated with a %PDF- header + ≥10 KB size). Do not re-implement OA fetching here.

Pass the verified candidate DOIs from references/library.bib:

ENGINE="${MEDSCI_SKILLS_ROOT:-$HOME/workspace/medsci-skills}/skills/fulltext-retrieval/fetch_oa.py"
# extract DOIs from references/library.bib → dois.txt (one per line)
python3 "$ENGINE" dois.txt -o pdfs/ -e <contact-email> --report pdfs/retrieval_report.json

For Zotero-resident PDFs and higher-yield, proxy-aware retrieval, use /lit-sync Phase 2.7, which also invokes /fulltext-retrieval and triggers Zotero's native "Find Available PDF".

Alternative sources (legitimate only)

For DOIs that open access cannot reach (listed in pdfs/manual_needed.txt):

• Institutional access / proxy / VPN — through your library's own subscriptions.
• Interlibrary loan (ILL) — request via library services.
• Author contact — email the corresponding author for a copy or preprint.

Never bypass paywalls or publisher access controls, and do not configure unauthorized PDF mirrors. Rate limits and PDF validation are handled inside /fulltext-retrieval.

Phase 6: Gap Analysis

When called during manuscript writing (especially by /write-paper Phase 7):

1. Read the manuscript to extract all inline citations.
2. Compare cited references against the search results.
3. Identify gaps:
   • Key papers in the field that are not cited.
   • Outdated references when newer versions exist.
   • Missing methodological references (e.g., statistical methods, reporting guidelines).
4. Report findings to the user with specific suggestions.

Specialized Search Modes

Mode: Manuscript Paper Reference Pool

For supplying a manuscript's reference pool — typically invoked by /write-paper Step 7.3c (or /self-review Phase 2.5c-2) when the reference adequacy gate finds the draft under target or a named method uncited, but usable directly when building out an original-research bibliography.

This mode is deliberately broad: for an original-research article, return 25–40 verified candidates, not the ~10 a quick search settles on. Do not stop early unless the field is genuinely sparse — and if it is, say so explicitly rather than returning a thin list silently. Respect a narrower journal reference cap or user scope when one is given.

Structure the pool across six candidate categories so the gaps the adequacy gate cares about are all covered:

1. Background / disease burden / clinical context — establishes why the question matters.
2. Gap-defining prior studies — the work the manuscript extends or contradicts.
3. Comparator / comparable-design cohorts — studies the Results will be measured against.
4. Methods / statistical canonical sources — the originating reference for every named method, model, score, equation, or diagnostic criterion (e.g. competing-risk model, multiple imputation, E-value, eGFR equation, concordance statistic). This is the category that clears Methods named-method gaps.
5. Reporting-guideline sources — STROBE, TRIPOD(+AI), CONSORT, PRISMA(-DTA), STARD, etc.
6. Interpretation / mechanism / limitation support — grounds Discussion claims.

For each candidate, report: PMID/DOI, verification status, candidate category, the target manuscript section it belongs in, and a one-line why it is needed.

Boundary (unchanged): every entry is API-verified before inclusion, and BibTeX is appended only to references/library.bib — the candidate pool for /lit-sync to import into Zotero. Never write to manuscript/_src/refs.bib; that SSOT belongs to /lit-sync. This mode produces candidates; it does not decide inclusion (the user does) and it does not insert references into the manuscript bib.

Mode: Crowding Check

Run before a study is designed, not after. The question is not "what has been written about this topic" — a background search answers that and still leaves the trap open. It is narrower and it is four questions:

Each gets its own verdict. A design can be original on one axis and fully occupied on another, and collapsing the four into one answer is what hides that.

Why the fourth row is not vanity: a design once matched an existing paper on frame, coding axis and target journal, and that paper was already published in the journal it was first choice for. A redesign on a different axis then turned out to be partly occupied too — three papers were already coding the same thing as a single item — which did not kill it but did change the claim that could honestly be made, from "nobody has looked at this" to "nobody has decomposed it by provenance". That is a real result of this mode: most of the time it narrows a claim rather than ending a project, and a narrowed claim survives review where the broad one would not.

Search the way a competitor would: the exact frame, the exact measure, and the journal's own site, not only the topic. Report the four verdicts and the papers behind each, then let the user decide. /design-study and /orchestrate should route here first when a new study is being scoped.

Mode: Systematic Search

For systematic reviews or comprehensive literature sections:

1. Document the full search strategy (PRISMA-compliant).
2. Record: database, date of search, query string, number of results.
3. Track inclusion/exclusion at each screening step.
4. Output a PRISMA flow diagram data summary.

Mode: Quick Cite

For quickly finding a single reference the user describes:

1. User says something like "that 2023 paper by Smith about AI in chest X-ray."
2. Search PubMed and Semantic Scholar with the described details.
3. Present top 3 candidates.
4. User confirms which one.
5. Generate BibTeX entry.

Mode: Related Papers

For expanding from a known paper:

1. User provides a PMID or DOI.
2. Use find_related_articles to get related papers.
3. Use Semantic Scholar for citation-based recommendations.
4. Present results ranked by relevance.

For a structured, dedup-aware, PRISMA-countable expansion (backward + forward + similar) prefer Phase 2.5: Citation Searching with references/snowball.py, which appends verified candidates to references/library.bib and reports a citation-searching count.

Mode: Embase Browser Automation

Embase has no public API. Use Chrome browser automation (MCP) to search and export:

1. Navigate to embase.com — institutional SSO authenticates automatically. If cookie error (login?error#), clear Elsevier/Embase cookies and retry.
2. Go to Advanced Search tab.
3. Enter Embase-syntax query (Emtree /exp + :ab,ti field tags). Uncheck "Map to preferred term in Emtree" when using explicit /exp terms.
4. After results appear, use "Select number of items" dropdown → select total count.
5. Click Export (in Results section) → choose CSV format → check fields: Title, Author names, Source, Publication year, Publication type, DOI, Abstract, Language of article, Medline PMID.
6. Click Export → Download tab opens → click Download.
7. CSV is in row format (records separated by blank rows) — parse with:

   # Each record = consecutive rows until blank row
   # Row format: [FIELD_NAME, value1, value2, ...]
   # AUTHOR NAMES row has multiple values (one per author)
   

PubMed → Embase query translation:

• MeSH [Mesh] → Emtree /exp
• [tiab] → :ab,ti
• [Title/Abstract] → :ab,ti
• Boolean operators stay the same (AND, OR)
• Phrase search: use single quotes in Embase ('artificial ascites')

Error Handling

• If a search returns 0 results, broaden the query (remove one concept or use broader MeSH terms) and retry.
• CrossRef HTTP errors (token-saving rules):
  • 403 (rate-limited): Do NOT retry. Skip CrossRef silently → verify via PubMed title search instead.
  • 303 (redirect): Follow the redirect if possible. If not, skip CrossRef → PubMed fallback.
  • Any repeated failure: After the first CrossRef 403/303 in a session, assume CrossRef is rate-limiting and skip CrossRef for ALL remaining references. Go directly to PubMed title verification. This avoids N×retry token waste.
  • Never print raw error messages like "Request failed with status code 403." Collect failures silently and report a single summary line at the end: CrossRef unavailable for {N} references (rate-limited). Verified via PubMed instead.
• If a DOI does not resolve via CrossRef (after applying the rules above), try searching PubMed by title to confirm the reference exists.
• If the user provides a reference that cannot be verified by any method, clearly state: "This reference could not be verified. Please check manually before submission."
• Never silently include an unverified reference.

What This Skill Does NOT Do

• Does not download from paywalled journals without user-provided credentials or institutional access.
• Does not assess the quality of evidence (use /analyze-stats or /check-reporting for that).
• Does not write the literature review text (use /write-paper for that).
• Does not fabricate any part of a citation.