continuous-learning

Continuous Learning

Summary

Continuous Learning skill domain covering knowledge operations.

Tools: High, Medium, Low, Negative, Attack Patterns, Defense Patterns, Tool Behaviors, Environment Patterns (+1 more)

Domain: knowledge

Use Cases

1. Post-Engagement Learning — After a penetration test, extract reusable patterns from observations and tool outputs
2. Cross-Session Knowledge Building — Accumulate tool behavior knowledge across multiple engagements into structured entries
3. Pattern Recognition — Identify recurring vulnerabilities, tool limitations, and effective technique combinations
4. Knowledge Confidence Tracking — Assign and update confidence scores for learned facts based on corroboration frequency
5. Memory Layering — Distill raw observations into layered knowledge: immediate notes → verified patterns → core principles

Activation

• After completing a penetration test engagement
• When encountering a novel attack technique or defense
• After a tool produces unexpected results
• When identifying recurring patterns across targets
• User says "learn", "remember this", "pattern", "lesson learned"

Learning Cycle

┌──────────┐    ┌───────────┐    ┌───────────┐
│ Pattern  │───→│ Extract & │───→│ Confidence │
│ Detection│    │ Structure │    │  Scoring  │
└──────────┘    └───────────┘    └───────────┘
                                        │
┌──────────┐    ┌───────────┐    ┌───────┴───┐
│ Cross-   │←──│  Memory   │←──│  Storage  │
│ Reference│    │ Retrieval │    │           │
└──────────┘    └───────────┘    └───────────┘

Step 1: Pattern Detection

Identify learnable patterns from observations:

Step 2: Extract & Structure

Transform raw observation into structured knowledge:

## Knowledge Entry: [ID]
- **Category:** [Attack / Defense / Tool / Environment / Engagement]
- **Context:** [When/where this was observed]
- **Pattern:** [What happened]
- **Root Cause:** [Why it happened, if known]
- **Applicability:** [When this knowledge is relevant]
- **Source:** [Engagement type, tool, or research]
- **Date:** [When observed]

Step 3: Confidence Scoring

Rate confidence based on supporting evidence:

Step 4: Storage

Save structured knowledge to the appropriate memory layer:

Short-term (Engagement): Tactical observations for current engagement

• Specific tool configurations that worked
• Target-specific quirks discovered
• Time-critical findings to revisit

Medium-term (Technique): Technique-level knowledge reusable across engagements

• Effective payload patterns for specific technologies
• Bypass techniques for specific WAF/IDS products
• Tool-specific tips and gotchas

Long-term (Strategic): Strategic patterns about security landscapes

• Emerging attack technique trends
• Common architectural weaknesses by industry
• Tool capability boundaries and gaps

Step 5: Cross-Reference

Link new knowledge to existing entries:

• Does this contradict a previous observation?
• Does this reinforce a previous observation?
• Does this fill a gap in existing knowledge?
• Is this a variant of a known pattern?

Knowledge Categories

Attack Pattern Library

### [Pattern Name]
- **Technique:** [ATT&CK technique ID if applicable]
- **Prerequisites:** [What must be true for this to work]
- **Steps:** [How to execute]
- **Indicators of success:** [How to know it worked]
- **Variations observed:** [Different contexts where it applied]
- **Counter-indications:** [When NOT to attempt this]

Tool Mastery Notes

### [Tool Name] - [Specific Use Case]
- **Command:** [Exact command with flags]
- **Context:** [When this configuration is optimal]
- **Output interpretation:** [How to read the results]
- **Gotchas:** [Common mistakes or misleading output]
- **Alternatives:** [Other tools for the same job]
- **Performance notes:** [Speed/resource considerations]

Engagement Lessons

### Lesson: [Title]
- **Engagement type:** [Black box / White box / Red team / Bug bounty]
- **What happened:** [Description]
- **What we learned:** [Key takeaway]
- **What we'd do differently:** [Process improvement]
- **Applicable scenarios:** [When this lesson is relevant]

Learning Triggers

Automatic pattern extraction should occur when:

1. An exploit succeeds unexpectedly — Why did it work? What was different?
2. A tool gives unexpected output — Is this a false positive, a real finding, or a tool bug?
3. A target behaves unusually — Custom defense? Misconfiguration? Honeypot?
4. An attack chain works particularly well — What made the chain effective?
5. A previously reliable technique fails — Has the target been patched? Is there a new defense?
6. Time is wasted on a dead end — What signal was missed that could have prevented this?

Integration with Other Skills

Knowledge Quality Rules

1. Never store assumptions as facts — Label speculation clearly
2. Always include context — Where, when, and under what conditions
3. Prefer specific over general — "SQL injection in parameter X of WordPress plugin Y v3.2" beats "SQL injection exists"
4. Include negative results — "Tool X missed finding Y" is as valuable as "Tool X found Y"
5. Date everything — Knowledge has a shelf life; dated entries can be refreshed
6. Source attribution — Every entry must trace back to a specific observation or research

Detection Methods

Learning Pipeline Anomalies

• Training data injection: Sudden spike in training data volume from untrusted source.
• Model weight drift: Model weights changing unexpectedly between training cycles.
• Backdoor activation: Specific input patterns triggering anomalous predictions.

RAG / Knowledge Base Indicators

• Vector store poisoning: Documents with embedding far from cluster centroid.
• Retrieval anomalies: Specific queries consistently retrieving same documents (potential trigger).
• Cross-user data leakage: User A's query retrieving User B's indexed documents.

SIEM Detection Rules

• Splunk SPL: index=ml training_run=* | stats dc(data_source) by model_version | where dc > 5
• LangSmith / Helicone: Anomaly detection on RAG retrieval patterns.

Defense Evasion Techniques

Data Poisoning Stealth

• Slow injection: Add poisoned samples over multiple training cycles; below distribution shift threshold.
• Mimic legitimate distribution: Match poisoned samples' statistics to legitimate data.
• Trigger-based backdoor: Activates only on specific input patterns; otherwise benign.
• Label-flipping attacks: Flip labels of small % of samples; degrades model accuracy subtly.

RAG Poisoning Stealth

• Match embedding distribution: Poisoned documents close to legitimate cluster centroid.
• Trigger via specific query: Backdoor activates only on attacker's specific query pattern.
• Cross-tenant indexing: Index poison in tenant A; affect queries from tenant B (if not isolated).

Memory Poisoning Stealth

• Long-term memory manipulation: Modify agent memory to plant triggers for future sessions.
• Cache poisoning: Poison LLM gateway cache; affects future users.
• Gradual trust building: Build up reputation over time, then exploit at scale.

Anti-Patterns

• Storing unverified findings — Verify before storing (use verification-loop)
• Over-generalizing — One observation does not make a universal rule
• Ignoring context — A technique that worked in one context may fail in another
• Knowledge hoarding — Store structured patterns, not raw data dumps
• Never pruning — Review and update stored knowledge periodically
• Confidence inflation — Be honest about confidence levels

Orchestration

ECC Loop Pattern

• Pattern: Learning Cycle (detect pattern → extract & structure → score confidence → store → cross-reference)
• Rationale: Learning is inherently iterative — each observation refines understanding, confidence scores evolve with more evidence, and cross-referencing with existing knowledge prevents contradictions
• Integration: All skills (consumes observations from every engagement), verification-loop (learns which verification methods are most reliable), search-first (learns which sources have best results per topic), terminal-ops (learns optimal evidence formats)

Cross-Skill Pipeline

[all skills] → continuous-learning → MEMORY.md / chronicle
                       ↓                         ↑
              verification-loop (FP patterns)   deep-research (context enrichment)

Quality Gate

• Pre-condition: Observation from real engagement or verified research
• Post-condition: Structured knowledge entry with confidence level and source attribution
• Verification: Entry cross-referenced against existing knowledge, no contradictions