# 🤖 Agentic Compaction & Pipeline System

> **Complete open-source implementation of Claude Code-level architecture for deterministic multi-agent orchestration**

A production-ready TypeScript implementation featuring:
- **Context Compaction** - Intelligent conversation summarization and token management
- **Deterministic Orchestration** - State machine controls flow, not LLM decisions
- **Parallel Execution** - Up to 12 concurrent agent sessions
- **Event-Driven Coordination** - Agents finish work → next step triggers automatically

---

## 📋 Table of Contents

- [Features](#-features)
- [Architecture](#-architecture)
- [Quick Start](#-quick-start)
- [Component Overview](#-component-overview)
- [Usage Examples](#-usage-examples)
- [API Reference](#-api-reference)
- [Integration](#-integration)
- [Download](#-download)

---

## ✨ Features

### 1. Context Compaction System

| Feature | Description |
|---------|-------------|
| Token Counting | Character-based approximation (~4 chars/token) |
| Conversation Summarization | LLM-powered intelligent summarization |
| Context Compaction | 4 strategies: sliding-window, summarize-old, priority-retention, hybrid |
| Budget Management | Track and manage token budgets |

### 2. Deterministic Pipeline System

| Feature | Description |
|---------|-------------|
| State Machine | Deterministic flow control (no LLM decisions) |
| Parallel Execution | 4 projects × 3 roles = 12 concurrent sessions |
| Event Bus | Pub/sub coordination between agents |
| Workspace Isolation | Per-agent tools, memory, identity, files |
| YAML Workflows | OpenClaw/Lobster-compatible definitions |

---

## 🏗️ Architecture

```
┌─────────────────────────────────────────────────────────────────────┐
│                        Pipeline Orchestrator                         │
│                                                                      │
│  ┌────────────────┐  ┌────────────────┐  ┌────────────────┐         │
│  │  State Machine │  │   Event Bus    │  │ Parallel Exec  │         │
│  │ (Deterministic)│  │ (Coordination) │  │ (Concurrency)  │         │
│  └────────────────┘  └────────────────┘  └────────────────┘         │
│                              │                                       │
│  ┌───────────────────────────┴───────────────────────────┐          │
│  │                    Agent Workspaces                    │          │
│  │                                                        │          │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐    │          │
│  │  │  Programmer │  │   Reviewer  │  │    Tester   │    │          │
│  │  │   (Opus)    │  │  (Sonnet)   │  │  (Sonnet)   │    │          │
│  │  │             │  │             │  │             │    │          │
│  │  │ • Tools     │  │ • Tools     │  │ • Tools     │    │          │
│  │  │ • Memory    │  │ • Memory    │  │ • Memory    │    │          │
│  │  │ • Workspace │  │ • Workspace │  │ • Workspace │    │          │
│  │  └─────────────┘  └─────────────┘  └─────────────┘    │          │
│  │                                                        │          │
│  └────────────────────────────────────────────────────────┘          │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────────────┐
│                       LLM Provider (ZAI SDK)                         │
└─────────────────────────────────────────────────────────────────────┘
```

---

## 🚀 Quick Start

### Prerequisites

```bash
# Install dependencies
bun add z-ai-web-dev-sdk
```

### Basic Usage

```typescript
import { 
  PipelineOrchestrator, 
  runWorkflow,
  ContextCompactor,
  TokenCounter 
} from './pipeline-system';

// Initialize orchestrator
const orchestrator = new PipelineOrchestrator();
await orchestrator.initialize();

// Create a code pipeline
const pipelineId = await orchestrator.createPipeline({
  name: 'Multi-Project Pipeline',
  projects: [
    {
      id: 'project-1',
      name: 'Authentication Module',
      tasks: [
        { type: 'implement', description: 'Create auth module', role: 'programmer' },
        { type: 'review', description: 'Review code', role: 'reviewer' },
        { type: 'test', description: 'Test implementation', role: 'tester' }
      ]
    }
  ],
  roles: ['programmer', 'reviewer', 'tester'],
  maxConcurrency: 12
});

// Or run a predefined workflow
const workflowId = await runWorkflow('code-pipeline', {
  projectId: 'my-project'
});
```

---

## 📦 Component Overview

### Agent System (`agent-system/`)

```
agent-system/
├── core/
│   ├── orchestrator.ts       # Agent lifecycle management
│   ├── token-counter.ts      # Token counting & budgeting
│   ├── context-manager.ts    # Context compaction logic
│   ├── subagent-spawner.ts   # Subagent creation & management
│   └── summarizer.ts         # LLM-powered summarization
├── agents/
│   ├── base-agent.ts         # Base agent class
│   └── task-agent.ts         # Task-specific agent
├── storage/
│   └── memory-store.ts       # Persistent storage
└── index.ts                  # Main exports
```

### Pipeline System (`pipeline-system/`)

```
pipeline-system/
├── core/
│   └── state-machine.ts      # Deterministic state machine
├── engine/
│   └── parallel-executor.ts  # Parallel execution engine
├── events/
│   └── event-bus.ts          # Event-driven coordination
├── workspace/
│   └── agent-workspace.ts    # Isolated agent workspaces
├── workflows/
│   └── yaml-workflow.ts      # YAML workflow parser
├── integrations/
│   └── claude-code.ts        # Claude Code integration
└── index.ts                  # Main exports
```

---

## 💡 Usage Examples

### Token Counting

```typescript
import { TokenCounter } from './agent-system';

const counter = new TokenCounter(128000);

// Count tokens in text
const result = counter.countText("Hello, world!");
console.log(result); // { tokens: 3, characters: 13, words: 2 }

// Count conversation
const conversation = [
  { role: 'user', content: 'Hello!' },
  { role: 'assistant', content: 'Hi there!' }
];
const budget = counter.getBudget(counter.countConversation(conversation).total);
console.log(budget);
// { used: 15, remaining: 123985, total: 124000, percentageUsed: 0.01 }
```

### Context Compaction

```typescript
import { ContextCompactor } from './agent-system';

const compactor = new ContextCompactor({
  maxTokens: 120000,
  strategy: 'hybrid',
  preserveRecentCount: 6
});

// Check if compaction needed
if (compactor.needsCompaction(messages)) {
  const result = await compactor.compact(messages);
  console.log(`Saved ${result.tokensSaved} tokens`);
  console.log(`Compression: ${(result.compressionRatio * 100).toFixed(1)}%`);
}
```

### State Machine

```typescript
import { DeterministicStateMachine } from './pipeline-system';

const definition = {
  id: 'code-pipeline',
  name: 'Code Pipeline',
  initial: 'start',
  states: {
    start: { type: 'start', onExit: [{ event: 'start', target: 'code' }] },
    code: { 
      type: 'action', 
      agent: 'programmer',
      onExit: [
        { event: 'completed', target: 'review' },
        { event: 'failed', target: 'failed' }
      ]
    },
    review: {
      type: 'choice',
      onExit: [
        { event: 'approved', target: 'end', condition: { type: 'equals', field: 'approved', value: true } },
        { event: 'rejected', target: 'code' }
      ]
    },
    end: { type: 'end' },
    failed: { type: 'end' }
  }
};

const sm = new DeterministicStateMachine(definition);
sm.start();
sm.sendEvent({ type: 'start', source: 'user', payload: {} });
```

### Parallel Execution

```typescript
import { ParallelExecutionEngine } from './pipeline-system';

const executor = new ParallelExecutionEngine({
  maxWorkers: 4,
  maxConcurrentPerWorker: 3
});

executor.start();

// Submit parallel tasks
const tasks = executor.submitBatch([
  { projectId: 'p1', role: 'programmer', type: 'implement', description: 'Auth', priority: 'high' },
  { projectId: 'p2', role: 'programmer', type: 'implement', description: 'Payment', priority: 'high' },
  { projectId: 'p3', role: 'programmer', type: 'implement', description: 'Dashboard', priority: 'medium' }
]);
```

### Event Bus

```typescript
import { EventBus, PipelineEventTypes } from './pipeline-system';

const eventBus = new EventBus();
eventBus.start();

// Subscribe to events
eventBus.subscribe({
  eventType: PipelineEventTypes.AGENT_COMPLETED,
  handler: async (event) => {
    console.log('Agent completed:', event.payload);
    // Trigger next step
    eventBus.publish({
      type: PipelineEventTypes.TASK_STARTED,
      source: 'orchestrator',
      payload: { nextAgent: 'reviewer' }
    });
  }
});
```

---

## 📚 API Reference

### PipelineOrchestrator

```typescript
class PipelineOrchestrator {
  // Initialize the system
  async initialize(): Promise<void>
  
  // Create a pipeline
  async createPipeline(config: PipelineConfig): Promise<string>
  
  // Create from YAML workflow
  async createPipelineFromYAML(workflowId: string, context?: object): Promise<string>
  
  // Get pipeline status
  getPipelineStatus(pipelineId: string): PipelineResult | undefined
  
  // Cancel pipeline
  async cancelPipeline(pipelineId: string): Promise<void>
  
  // Subscribe to events
  onEvent(eventType: string, handler: Function): () => void
  
  // Get statistics
  getStats(): object
  
  // Shutdown
  async shutdown(): Promise<void>
}
```

### Quick Start Functions

```typescript
// Create simple code pipeline
createCodePipeline(projects: ProjectConfig[]): Promise<string>

// Create parallel pipeline
createParallelPipeline(config: PipelineConfig): Promise<string>

// Run predefined workflow
runWorkflow(workflowId: string, context?: object): Promise<string>
```

---

## 🔗 Integration

### With Claude Code

```typescript
import { PipelineOrchestrator } from './pipeline-system';

const orchestrator = new PipelineOrchestrator();
await orchestrator.initialize();

// Use in Claude Code environment
const pipelineId = await orchestrator.createPipeline({
  name: 'Claude Code Pipeline',
  projects: [
    {
      id: 'claude-project',
      name: 'Claude Integration',
      tasks: [
        { type: 'implement', description: 'Add MCP server', role: 'programmer' },
        { type: 'review', description: 'Review changes', role: 'reviewer' },
        { type: 'test', description: 'Test integration', role: 'tester' }
      ]
    }
  ]
});
```

### With OpenClaw

```typescript
import { runWorkflow, WorkflowRegistry } from './pipeline-system';

// Register custom workflow
const registry = new WorkflowRegistry();
registry.register({
  id: 'custom-openclaw-workflow',
  name: 'Custom Workflow',
  initial: 'start',
  states: { /* ... */ }
});

// Run workflow
await runWorkflow('custom-openclaw-workflow', {
  projectId: 'my-project'
});
```

### Lobster-Compatible YAML

```yaml
id: code-pipeline
name: Code Pipeline
initial: start
states:
  start:
    type: start
    on:
      start: code

  code:
    type: action
    role: programmer
    timeout: 30m
    retry:
      maxAttempts: 2
      backoff: exponential
    on:
      completed: review
      failed: failed

  review:
    type: choice
    on:
      approved: test
      rejected: code

  test:
    type: action
    role: tester
    on:
      passed: end
      failed: failed

  end:
    type: end

  failed:
    type: end
```

---

## 📥 Download

Pre-built packages available:

| Package | Size | Contents |
|---------|------|----------|
| `complete-agent-pipeline-system.zip` | 60KB | Both systems + docs |
| `agent-system.zip` | 27KB | Context & memory management |
| `pipeline-system.zip` | 29KB | Deterministic orchestration |

---

## 📋 Predefined Workflows

| Workflow ID | Description |
|-------------|-------------|
| `code-pipeline` | Code → Review → Test (max 3 review iterations) |
| `parallel-projects` | Run multiple projects in parallel |
| `human-approval` | Workflow with human approval gates |

---

## 🎯 Key Principles

1. **Deterministic Flow**: State machines control the pipeline, not LLM decisions
2. **Event-Driven**: Agents communicate through events, enabling loose coupling
3. **Parallel Execution**: Multiple agents work concurrently with resource isolation
4. **Workspace Isolation**: Each agent has its own tools, memory, and file space
5. **YAML Workflows**: Define pipelines declaratively, compatible with Lobster

---

## 📄 License

MIT License - Free to use, modify, and distribute.

---

## 🤝 Contributing

Contributions welcome! This system is designed for easy integration with:
- Claude Code
- OpenClaw
- Lobster
- Custom agent frameworks

---

## 📊 Project Statistics

- **Total Files**: 32 source files
- **Total Code**: ~100KB of TypeScript
- **Components**: 6 major modules
- **Predefined Workflows**: 3

---

Built with ❤️ for the AI agent community.