# Agentic Compaction & Pipeline System

<p align="center">
  <strong>A comprehensive open-source implementation of context compaction mechanisms<br>
  and deterministic multi-agent pipeline orchestration</strong>
</p>

<p align="center">
  <em>Designed for seamless integration with Claude Code, OpenClaw, and custom AI systems</em>
</p>

---

<p align="center">
  <a href="#features">Features</a> •
  <a href="#installation">Installation</a> •
  <a href="#quick-start">Quick Start</a> •
  <a href="#integrations">Integrations</a> •
  <a href="#delegation-system">Delegation System</a> •
  <a href="#api-reference">API Reference</a>
</p>

---

## 🤖 About This Project

<div align="center">

### ⚡ This project was 100% autonomously built by AI ⚡

**Z.AI GLM-5** created this entire codebase in a single session — designing the architecture, 
implementing all modules, writing comprehensive documentation, packaging the releases, 
and even pushing everything to this Git repository.

**[Learn more about Z.AI GLM-5 →](https://z.ai/subscribe?ic=R0K78RJKNW)**

*Yes, the README you're reading right now was written by the AI too! 🎉*

</div>

---

## Overview

This project provides three complementary systems:

1. **Agent System** - Context compaction, token management, and agent orchestration
2. **Pipeline System** - Deterministic state machine, parallel execution, and event-driven coordination
3. **Delegation System** - Intelligent request routing, agent pool management, and busy-state handling

Built based on the architectural principles described in [How I Built a Deterministic Multi-Agent Dev Pipeline Inside OpenClaw](https://dev.to/ggondim/how-i-built-a-deterministic-multi-agent-dev-pipeline-inside-openclaw-and-contributed-a-missing-4ool).

---

## ✨ Features

### Agent System

- ✅ **Token Counting & Management** - Accurate token estimation with budget tracking
- ✅ **Context Compaction** - 4 strategies: sliding-window, summarize-old, priority-retention, hybrid
- ✅ **Conversation Summarization** - LLM-powered summarization with key points extraction
- ✅ **Agent Orchestration** - Lifecycle management, task routing, event handling
- ✅ **Subagent Spawning** - 6 predefined subagent types for task delegation
- ✅ **Persistent Storage** - File-based memory store for agent state
- ✅ **Claude Code Integration** - Full support for Claude Code CLI/IDE
- ✅ **OpenClaw Integration** - Native integration with OpenClaw workflows

### Pipeline System

- ✅ **Deterministic State Machine** - Flow control without LLM decisions
- ✅ **Parallel Execution Engine** - Worker pools with concurrent agent sessions
- ✅ **Event-Driven Coordination** - Pub/sub event bus with automatic trigger chains
- ✅ **Workspace Isolation** - Per-agent tools, memory, identity, permissions
- ✅ **YAML Workflow Parser** - Lobster-compatible workflow definitions

### 🆕 Delegation System (NEW!)

- ✅ **Request Classification** - Fast analysis (<50ms) for optimal routing
- ✅ **Agent Pool Management** - Auto-scaling pool with 8 agent types
- ✅ **Delegation Engine** - 4 strategies: full, parallel, hierarchical, hybrid
- ✅ **Progress Streaming** - Real-time updates with SSE/WebSocket support
- ✅ **Context Handoff Protocol** - Seamless inter-agent context transfer
- ✅ **Quality Gate** - Confidence thresholds and auto-escalation
- ✅ **3rd Party Adapters** - Native support for OpenClaw, Claude Code CLI, Cursor, Aider

---

## Architecture

```
┌─────────────────────────────────────────────────────────────────────────┐
│                      AGENTIC PIPELINE SYSTEM                            │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐         │
│  │  AGENT SYSTEM   │  │ PIPELINE SYSTEM │  │DELEGATION SYSTEM│         │
│  ├─────────────────┤  ├─────────────────┤  ├─────────────────┤         │
│  │ • Token Counter │  │ • State Machine │  │ • Classifier    │         │
│  │ • Context Mgr   │  │ • Parallel Exec │  │ • Pool Manager  │         │
│  │ • Summarizer    │  │ • Event Bus     │  │ • Deleg. Engine │         │
│  │ • Orchestrator  │  │ • Workspace     │  │ • Progress Str. │         │
│  │ • Subagent Sp.  │  │ • YAML Workflow │  │ • Quality Gate  │         │
│  │ • Memory Store  │  │ • Integration   │  │ • Handoff Proto │         │
│  │ ───────────────  │  └─────────────────┘  └─────────────────┘         │
│  │ INTEGRATIONS:    │                                                │
│  │ • Claude Code ✅ │                                                │
│  │ • OpenClaw ✅    │                                                │
│  └─────────────────┘                                                │
│                                                                         │
│  ┌─────────────────────────────────────────────────────────────────┐   │
│  │                      INTEGRATION LAYER                           │   │
│  │  Claude Code │ OpenClaw │ Cursor │ Aider │ Copilot │ Custom     │   │
│  └─────────────────────────────────────────────────────────────────┘   │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘
```

### Delegation Flow (Busy-State Handling)

```
User Request → Request Classifier (<50ms)
                    │
                    ▼
            ┌───────────────┐
            │ Classification │
            │ • Complexity   │
            │ • Agent Type   │
            │ • Priority     │
            └───────────────┘
                    │
                    ▼
            ┌───────────────┐
            │ Pool Manager  │
            │ • Find Agent  │
            │ • Scale Pool  │
            └───────────────┘
                    │
        ┌───────────┴───────────┐
        ▼                       ▼
┌───────────────┐       ┌───────────────┐
│   Delegate    │       │  Queue/Wait   │
│   to Agent    │       │  (if busy)    │
└───────────────┘       └───────────────┘
        │                       │
        ▼                       ▼
┌───────────────┐       ┌───────────────┐
│  Progress     │       │  Escalate     │
│  Streaming    │       │  to Main      │
└───────────────┘       └───────────────┘
        │                       │
        └───────────┬───────────┘
                    ▼
            ┌───────────────┐
            │ Quality Gate  │
            │ • Validate    │
            │ • Confidence  │
            └───────────────┘
                    │
                    ▼
              Response
```

---

## Installation

### Prerequisites

- Node.js 18+ or Bun
- TypeScript 5+

### From Source

```bash
# Clone the repository
git clone https://github.rommark.dev/admin/Agentic-Compaction-and-Pipleline-by-GLM-5.git
cd Agentic-Compaction-and-Pipleline-by-GLM-5

# Install dependencies
bun install

# Run tests
bun run delegation-system/test.ts
```

### Using Zip Packages

Download the appropriate package from the `downloads/` directory:

| Package | Description | Use Case |
|---------|-------------|----------|
| `agent-system.zip` | Context compaction & orchestration | Building custom AI agents |
| `pipeline-system.zip` | Deterministic pipelines | Multi-agent workflows |
| `delegation-system.zip` | Busy-state handling | High-traffic AI services |
| `complete-agent-pipeline-system.zip` | Full system | Complete integration |

---

## Quick Start

### Delegation System (Busy-State Handling)

```typescript
import { DelegationSystem } from './delegation-system';

// Initialize the system
const system = new DelegationSystem();
await system.initialize();

// Process a request (auto-delegates when busy)
const response = await system.process({
  id: 'req-1',
  content: 'Review this code for security issues',
  type: 'review',
  streamProgress: true
});

console.log(response.result);
console.log(`Processed by: ${response.agentsUsed.join(', ')}`);
console.log(`Confidence: ${response.confidence}`);

// Subscribe to progress updates
system.onProgress('req-1', (event) => {
  console.log(`[${event.type}] ${event.message}`);
});
```

### Agent System

```typescript
import { ContextManager, TokenCounter, Summarizer } from './agent-system';

// Initialize components
const tokenCounter = new TokenCounter(128000);
const summarizer = new Summarizer();
const contextManager = new ContextManager(tokenCounter, summarizer, {
  maxTokens: 100000,
  compactionStrategy: 'hybrid',
  reserveTokens: 20000
});

// Add messages with auto-compaction
contextManager.addMessage({ role: 'user', content: 'Hello!' });

if (contextManager.needsCompaction()) {
  const result = await contextManager.compact();
  console.log(`Saved ${result.tokensSaved} tokens`);
}
```

### Pipeline System

```typescript
import { DeterministicStateMachine, ParallelExecutionEngine } from './pipeline-system';

const executor = new ParallelExecutionEngine({ maxConcurrency: 12 });
const results = await executor.executeAll([
  { id: 1, type: 'analyze', data: { target: 'module-a' } },
  { id: 2, type: 'analyze', data: { target: 'module-b' } }
]);
```

---

## Integrations

### 3rd Party AI Coding Tools Integration

The delegation system provides **native integration** for popular AI coding tools:

#### OpenClaw Integration

```typescript
import { DelegationSystem } from './delegation-system';

const system = new DelegationSystem();
await system.initialize();

// Get OpenClaw adapter
const openclaw = system.getAdapter('openclaw');

// Configure for OpenClaw pattern (4 projects × 3 roles)
await openclaw.initialize({
  pool: { maxParallelAgents: 12 },
  delegation: { autoDelegate: true }
});

// Process with OpenClaw-compatible workflow
const response = await system.process({
  content: 'Analyze security across all modules',
  metadata: { integration: 'openclaw' }
});
```

#### Claude Code CLI Integration

```typescript
import { DelegationSystem } from './delegation-system';

const system = new DelegationSystem();
await system.initialize();

const claudeCode = system.getAdapter('claude-code');

// Get Claude Code compatible tools
const tools = claudeCode.getTools();
// [
//   { name: 'delegate_task', ... },
//   { name: 'check_pool_status', ... }
// ]

// Process with context compaction
const response = await system.process({
  content: 'Implement user authentication',
  metadata: { integration: 'claude-code' }
});
```

#### Custom Integration

```typescript
import { DelegationSystem } from './delegation-system';

const system = new DelegationSystem();

// Register custom adapter
await system.registerIntegration('my-tool', {
  delegation: { enabled: true },
  pool: { maxSize: 20 }
});

// Use the custom integration
const response = await system.process({
  content: 'Process this request',
  metadata: { integration: 'my-tool' }
});
```

---

## Delegation System

### Agent Types

| Type | Purpose | Timeout | Concurrency |
|------|---------|---------|-------------|
| `fast-responder` | Quick answers, status checks | 5s | 10 |
| `explorer` | Code navigation, file search | 30s | 4 |
| `researcher` | Deep analysis, documentation | 60s | 3 |
| `coder` | Code generation, implementation | 45s | 4 |
| `reviewer` | Code review, quality checks | 30s | 3 |
| `planner` | Architecture, planning | 60s | 2 |
| `executor` | Command execution, file ops | 30s | 3 |
| `analyzer` | Debugging, profiling | 45s | 3 |

### Delegation Strategies

| Strategy | Description | When Used |
|----------|-------------|-----------|
| `full` | Complete delegation to single agent | Quick tasks, simple requests |
| `parallel` | Multiple agents work together | Multi-file, high complexity |
| `hierarchical` | Main agent + subagent collaboration | Complex, needs oversight |
| `hybrid` | Combination based on context | Streaming, progressive results |

### Request Classification

Requests are classified into 4 complexity levels:

| Level | Response Time | Example |
|-------|--------------|---------|
| `quick` | < 2s | "What is TypeScript?" |
| `moderate` | 2-15s | "Review this function for bugs" |
| `streaming` | 5-60s | "Analyze the entire codebase" |
| `complex` | 15-120s | "Implement a new authentication system" |

---

## API Reference

### Delegation System API

```typescript
class DelegationSystem {
  // Initialization
  initialize(): Promise<void>;
  registerIntegration(type: IntegrationType, config?): Promise<Adapter>;
  
  // Main operations
  process(request: DelegationRequest): Promise<DelegationResponse>;
  quickClassify(content: string): { complexity: string; agent: AgentType };
  
  // Progress streaming
  onProgress(requestId: string, callback: (event: ProgressEvent) => void): () => void;
  onAllProgress(callback: (event: ProgressEvent) => void): () => void;
  
  // Status
  getStatus(): SystemStatus;
  getPoolStats(): PoolStats;
  isReady(): boolean;
  
  // Lifecycle
  shutdown(): Promise<void>;
  reset(): Promise<void>;
}
```

### Integration Adapters API

```typescript
interface IntegrationAdapter {
  type: IntegrationType;
  name: string;
  version: string;
  
  initialize(config: any): Promise<void>;
  shutdown(): Promise<void>;
  classifyRequest(request: any): Promise<RequestClassification>;
  delegateRequest(request: any, decision: DelegationDecision): Promise<DelegationResult>;
  streamProgress(requestId: string, callback: (event: ProgressEvent) => void): void;
  getStatus(): IntegrationStatus;
  getCapabilities(): string[];
}
```

---

## Examples

### Example 1: Busy-State Handling

```typescript
import { DelegationSystem } from './delegation-system';

const system = new DelegationSystem();
await system.initialize();

// Simulate high load
for (let i = 0; i < 20; i++) {
  system.process({
    content: `Task ${i}`,
    type: 'code'
  });
}

// This request will be delegated to available agent or queued
const response = await system.process({
  content: 'Urgent: Fix security vulnerability',
  type: 'debug',
  priority: 'critical'
});

// Check pool status
const stats = system.getPoolStats();
console.log(`Pool: ${stats.busyCount} busy, ${stats.idleCount} idle`);
```

### Example 2: Progress Streaming

```typescript
const system = new DelegationSystem();
await system.initialize();

const requestId = 'analysis-1';

// Subscribe to progress
const unsubscribe = system.onProgress(requestId, (event) => {
  switch (event.type) {
    case 'acknowledgment':
      console.log('🤖 Got it!');
      break;
    case 'delegation':
      console.log(`📋 Delegated to ${event.data?.agentType}`);
      break;
    case 'progress':
      console.log(`⏳ ${event.progress}% - ${event.message}`);
      break;
    case 'completion':
      console.log('✅ Done!');
      break;
  }
});

// Process with streaming
await system.process({
  id: requestId,
  content: 'Analyze entire codebase for security issues',
  type: 'analysis',
  streamProgress: true
});

unsubscribe();
```

### Example 3: Multi-Agent Parallel Execution

```typescript
const system = new DelegationSystem();
await system.initialize();

// OpenClaw pattern: 4 projects × 3 roles
const projects = ['frontend', 'backend', 'mobile', 'docs'];
const roles = ['security', 'performance', 'quality'];

const results = await Promise.all(
  projects.flatMap(project =>
    roles.map(role =>
      system.process({
        content: `Analyze ${project} for ${role} issues`,
        type: 'analysis',
        metadata: { project, role }
      })
    )
  )
);

console.log(`Completed ${results.length} parallel analyses`);
```

---

## Project Structure

```
├── agent-system/                 # Context compaction system
│   ├── core/                     # Core components
│   ├── agents/                   # Agent implementations
│   ├── integrations/             # Claude Code & OpenClaw integrations
│   ├── storage/                  # Persistent storage
│   └── utils/                    # Utilities
│
├── pipeline-system/              # Deterministic pipeline system
│   ├── core/                     # State machine
│   ├── engine/                   # Parallel execution
│   ├── events/                   # Event bus
│   ├── workspace/                # Workspace isolation
│   └── workflows/                # YAML parser
│
├── delegation-system/            # 🆕 Delegation system
│   ├── core/                     # Types, classifier, engine, handoff
│   ├── pool/                     # Agent pool management
│   ├── streaming/                # Progress streaming
│   ├── quality/                  # Quality gates
│   ├── integrations/             # 3rd party adapters
│   ├── test.ts                   # Test suite
│   └── index.ts                  # Main exports
│
├── examples/                     # Usage examples
├── downloads/                    # Zip packages
├── package.json
├── tsconfig.json
├── LICENSE
├── CHANGELOG.md
└── README.md
```

---

## Contributing

Contributions are welcome! Please read our contributing guidelines before submitting PRs.

1. Fork the repository
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

---

## License

MIT License - see [LICENSE](LICENSE) for details.

---

## Acknowledgments

- Inspired by [OpenClaw](https://github.com/ggondim/openclaw) and [Lobster](https://github.com/ggondim/lobster)
- Architectural patterns from [How I Built a Deterministic Multi-Agent Dev Pipeline](https://dev.to/ggondim/how-i-built-a-deterministic-multi-agent-dev-pipeline-inside-openclaw-and-contributed-a-missing-4ool)
- Claude Code integration patterns from Anthropic's documentation

---

## Support

For issues and feature requests, please use the [GitHub Issues](https://github.rommark.dev/admin/Agentic-Compaction-and-Pipleline-by-GLM-5/issues) page.

---

<div align="center">

**Built with ❤️ by [Z.AI GLM-5](https://z.ai/subscribe?ic=R0K78RJKNW)**

*100% Autonomous AI Development*

</div>