# Spark Intelligence Integration Guide for QwenClaw

## 🚀 Why Spark Intelligence?

**Spark Intelligence transforms QwenClaw from a stateless executor into a learning system that:**

- ✅ **Remembers** what worked and what didn't
- ✅ **Warns** before repeating mistakes
- ✅ **Promotes** validated wisdom automatically
- ✅ **Adapts** to your specific workflows
- ✅ **Improves** continuously through outcome tracking

---

## 📦 Installation

### Step 1: Install Spark Intelligence

**Windows (PowerShell):**
```powershell
irm https://raw.githubusercontent.com/vibeforge1111/vibeship-spark-intelligence/main/install.ps1 | iex
```

**Mac/Linux:**
```bash
curl -fsSL https://raw.githubusercontent.com/vibeforge1111/vibeship-spark-intelligence/main/install.sh | bash
```

**Manual Install:**
```bash
git clone https://github.com/vibeforge1111/vibeship-spark-intelligence
cd vibeship-spark-intelligence
python -m venv .venv
.venv\Scripts\activate  # Windows
# or: source .venv/bin/activate  # Mac/Linux
python -m pip install -e .[services]
```

### Step 2: Verify Installation

```bash
python -m spark.cli health
python -m spark.cli up
python -m spark.cli learnings
```

---

## 🔗 Integration with QwenClaw

### Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                     QwenClaw Session                        │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│              Spark Event Capture (Hooks)                    │
│  - PreToolUse   - PostToolUse   - UserPromptSubmit          │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│              Spark Intelligence Pipeline                    │
│  Capture → Distill → Transform → Store → Act → Guard → Learn│
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│           Pre-Tool Advisory (Before QwenClaw Acts)          │
│  BLOCK (0.95+) | WARNING (0.80-0.95) | NOTE (0.48-0.80)     │
└─────────────────────────────────────────────────────────────┘
```

### Configuration

Create `~/.spark/config.yaml`:

```yaml
spark:
  enabled: true
  session_id: qwenclaw-${timestamp}
  
  hooks:
    pre_tool_use: true
    post_tool_use: true
    user_prompt: true
    
  advisory:
    enabled: true
    min_score: 0.48
    cooldown_seconds: 300
    authority_levels:
      block: 0.95
      warning: 0.80
      note: 0.48
    
  memory:
    auto_capture: true
    min_importance: 0.55
    importance_boosts:
      causal_language: 0.15
      quantitative_data: 0.30
      technical_specificity: 0.15
    
  observatory:
    enabled: true
    sync_interval_seconds: 120
    vault_path: ~/Documents/Obsidian Vault/Spark-Intelligence-Observatory
```

### Start Spark + QwenClaw

```bash
# Terminal 1: Start Spark pipeline
python -m spark.cli up

# Terminal 2: Start QwenClaw
qwenclaw start

# Terminal 3: Send tasks
qwenclaw send "Refactor the authentication module"
```

---

## 🧠 How Spark Improves QwenClaw

### 1. Pre-Tool Advisory Guidance

**Before QwenClaw executes a tool**, Spark surfaces relevant lessons:

#### BLOCK Example (0.95+ score)
```
⚠️ BLOCKED: Spark advisory

Action: rm -rf ./node_modules
Reason: This command will delete critical dependencies.
        Last 3 executions resulted in 2+ hour rebuild times.
        Alternative: npm clean-install

Confidence: 0.97 | Validated: 12 times
```

#### WARNING Example (0.80-0.95 score)
```
⚠️ WARNING: Spark advisory

Action: Edit file without reading
File: src/config/database.ts
Pattern: This pattern failed 4 times in the last 24 hours.
         Missing context caused incorrect modifications.

Suggestion: Read the file first, then edit.
Reliability: 0.91 | Validated: 8 times
```

#### NOTE Example (0.48-0.80 score)
```
ℹ️ NOTE: Spark advisory

User Preference: Always use --no-cache flag for Docker builds
Context: Prevents stale layer caching issues
Captured from: Session #4521, 2 days ago
```

### 2. Anti-Pattern Detection

Spark identifies and corrects problematic workflows:

| Pattern | Detection | Correction |
|---------|-----------|------------|
| Edit without Read | File modified without prior read | Suggests reading first |
| Recurring Command Failures | Same command fails 3+ times | Suggests alternatives |
| Missing Tests | Code committed without tests | Reminds testing policy |
| Hardcoded Secrets | Secrets detected in code | Blocks and warns |

### 3. Memory Capture with Intelligence

**Automatic Importance Scoring (0.0-1.0):**

| Score | Action | Example |
|-------|--------|---------|
| ≥0.65 | Auto-save | "Remember: always use --no-cache for Docker" |
| 0.55-0.65 | Suggest | "I prefer TypeScript over JavaScript" |
| <0.55 | Ignore | Generic statements, noise |

**Signals that boost importance:**
- Causal language: "because", "leads to" (+0.15-0.30)
- Quantitative data: "reduced from 4.2s to 1.6s" (+0.30)
- Technical specificity: real tools, libraries, patterns (+0.15-0.30)

### 4. Auto-Promotion to Project Files

High-reliability insights automatically promote to:

**CLAUDE.md** - Wisdom, reasoning, context insights:
```markdown
## Docker Best Practices
- Always use `--no-cache` flag for production builds
- Validated: 12 times | Reliability: 0.96
```

**AGENTS.md** - Meta-learning, self-awareness:
```markdown
## Project Preferences
- Prefer TypeScript over JavaScript for large projects
- Test-first development required for core modules
```

**SOUL.md** - Communication preferences, user understanding:
```markdown
## User Communication Style
- Prefers concise explanations with code examples
- Values performance metrics and quantitative data
```

### 5. EIDOS Episodic Intelligence

Extracts structured rules from experience:

| Type | Description | Example |
|------|-------------|---------|
| **Heuristics** | General rules of thumb | "Always test before deploying" |
| **Sharp Edges** | Things to watch out for | "API rate limits hit at 100 req/min" |
| **Anti-Patterns** | What not to do | "Don't edit config without backup" |
| **Playbooks** | Proven approaches | "Database migration checklist" |
| **Policies** | Enforced constraints | "Must have tests for core modules" |

---

## 📊 Obsidian Observatory

Spark auto-generates **465+ markdown pages** with live Dataview queries:

### Generate Observatory

```bash
python scripts/generate_observatory.py --force --verbose
```

**Vault Location:** `~/Documents/Obsidian Vault/Spark-Intelligence-Observatory`

### What's Included

- **Pipeline Health** - 12-stage pipeline detail pages with metrics
- **Cognitive Insights** - Stored insights with reliability scores
- **EIDOS Episodes** - Pattern distillations and heuristics
- **Advisory Decisions** - Pre-tool guidance history
- **Explorer Views** - Real-time data exploration
- **Canvas View** - Spatial pipeline visualization

### Auto-Sync

Observatory syncs every **120 seconds** when pipeline is running.

---

## 📈 Measurable Outcomes

### Advisory Source Effectiveness

| Source | What It Provides | Effectiveness |
|--------|-----------------|---------------|
| **Cognitive** | Validated session insights | ~62% (dominant) |
| **Bank** | User memory banks | ~10% |
| **EIDOS** | Pattern distillations | ~5% |
| **Baseline** | Static rules | ~5% |
| **Trigger** | Event-specific rules | ~5% |
| **Semantic** | BM25 + embedding retrieval | ~3% |

### Timeline to Value

| Time | What Happens |
|------|--------------|
| **Hour 1** | Spark starts capturing events |
| **Hour 2-4** | Patterns emerge (tool effectiveness, error patterns) |
| **Day 1-2** | Insights get promoted to project files |
| **Week 1+** | Advisory goes live with pre-tool guidance |

---

## 🔧 CLI Commands

### Spark Commands

```bash
# Start pipeline
python -m spark.cli up

# Stop pipeline
python -m spark.cli down

# Check status
python -m spark.cli status

# View learnings
python -m spark.cli learnings

# View advisories
python -m spark.cli advisories

# Promote insight manually
python -m spark.cli promote <insight-id>

# Health check
python -m spark.cli health
```

### QwenClaw Commands

```bash
# Start QwenClaw
qwenclaw start

# Send task (Spark captures automatically)
qwenclaw send "Refactor the authentication module"

# Check status
qwenclaw status
```

---

## 🎯 Best Practices

### 1. Let Spark Learn Naturally
Just use QwenClaw normally. Spark captures and learns in the background.

### 2. Provide Explicit Feedback
Tell Spark what to remember:
```
"Remember: always use --force for this legacy package"
"I prefer yarn over npm in this project"
"Test files should be in __tests__ directory"
```

### 3. Review Advisories
Pay attention to pre-tool warnings. They're based on validated patterns.

### 4. Check Observatory
Review the Obsidian vault weekly to understand what Spark has learned.

### 5. Promote High-Value Insights
Manually promote insights that are immediately valuable:
```bash
python -m spark.cli promote <insight-id>
```

---

## 🚨 Troubleshooting

### Spark Not Capturing Events

**Check:**
```bash
python -m spark.cli health
python -m spark.cli status
```

**Solution:**
- Ensure Spark pipeline is running: `python -m spark.cli up`
- Verify hooks are enabled in config
- Check QwenClaw session ID matches

### Advisories Not Surfacing

**Check:**
```bash
python -m spark.cli advisories
```

**Solution:**
- Verify advisory min_score in config (default: 0.48)
- Check cooldown period (default: 300 seconds)
- Ensure insights have been validated (5+ times)

### Observatory Not Syncing

**Check:**
```bash
python scripts/generate_observatory.py --verbose
```

**Solution:**
- Verify Obsidian vault path in config
- Ensure vault exists
- Check sync interval (default: 120 seconds)

---

## 📚 Resources

- **Spark Docs:** https://spark.vibeship.co
- **GitHub:** https://github.com/vibeforge1111/vibeship-spark-intelligence
- **Onboarding:** `docs/SPARK_ONBOARDING_COMPLETE.md`
- **Quickstart:** `docs/QUICKSTART.md`
- **Obsidian Guide:** `docs/OBSIDIAN_OBSERVATORY_GUIDE.md`

---

## ✨ Summary

**Spark Intelligence + QwenClaw = Self-Evolving AI Assistant**

| Without Spark | With Spark |
|---------------|------------|
| Stateless execution | Continuous learning |
| Repeats mistakes | Warns before errors |
| No memory | Captures preferences |
| Static behavior | Evolves over time |
| No observability | Full Obsidian vault |

**Install Spark today and transform QwenClaw into a learning system!** 🧠✨