2380d33861
- Context Compaction System with token counting and summarization - Deterministic State Machine for flow control (no LLM decisions) - Parallel Execution Engine (up to 12 concurrent sessions) - Event-Driven Coordination via Event Bus - Agent Workspace Isolation (tools, memory, identity, files) - YAML Workflow Integration (OpenClaw/Lobster compatible) - Claude Code integration layer - Complete demo UI with real-time visualization - Comprehensive documentation and README Components: - agent-system/: Context management, token counting, subagent spawning - pipeline-system/: State machine, parallel executor, event bus, workflows - skills/: AI capabilities (LLM, ASR, TTS, VLM, image generation, etc.) - src/app/: Next.js demo application Total: ~100KB of production-ready TypeScript code
494 lines
14 KiB
Markdown
Executable File
494 lines
14 KiB
Markdown
Executable File
# 🤖 Agentic Compaction & Pipeline System
|
||
|
||
> **Complete open-source implementation of Claude Code-level architecture for deterministic multi-agent orchestration**
|
||
|
||
A production-ready TypeScript implementation featuring:
|
||
- **Context Compaction** - Intelligent conversation summarization and token management
|
||
- **Deterministic Orchestration** - State machine controls flow, not LLM decisions
|
||
- **Parallel Execution** - Up to 12 concurrent agent sessions
|
||
- **Event-Driven Coordination** - Agents finish work → next step triggers automatically
|
||
|
||
---
|
||
|
||
## 📋 Table of Contents
|
||
|
||
- [Features](#-features)
|
||
- [Architecture](#-architecture)
|
||
- [Quick Start](#-quick-start)
|
||
- [Component Overview](#-component-overview)
|
||
- [Usage Examples](#-usage-examples)
|
||
- [API Reference](#-api-reference)
|
||
- [Integration](#-integration)
|
||
- [Download](#-download)
|
||
|
||
---
|
||
|
||
## ✨ Features
|
||
|
||
### 1. Context Compaction System
|
||
|
||
| Feature | Description |
|
||
|---------|-------------|
|
||
| Token Counting | Character-based approximation (~4 chars/token) |
|
||
| Conversation Summarization | LLM-powered intelligent summarization |
|
||
| Context Compaction | 4 strategies: sliding-window, summarize-old, priority-retention, hybrid |
|
||
| Budget Management | Track and manage token budgets |
|
||
|
||
### 2. Deterministic Pipeline System
|
||
|
||
| Feature | Description |
|
||
|---------|-------------|
|
||
| State Machine | Deterministic flow control (no LLM decisions) |
|
||
| Parallel Execution | 4 projects × 3 roles = 12 concurrent sessions |
|
||
| Event Bus | Pub/sub coordination between agents |
|
||
| Workspace Isolation | Per-agent tools, memory, identity, files |
|
||
| YAML Workflows | OpenClaw/Lobster-compatible definitions |
|
||
|
||
---
|
||
|
||
## 🏗️ Architecture
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────┐
|
||
│ Pipeline Orchestrator │
|
||
│ │
|
||
│ ┌────────────────┐ ┌────────────────┐ ┌────────────────┐ │
|
||
│ │ State Machine │ │ Event Bus │ │ Parallel Exec │ │
|
||
│ │ (Deterministic)│ │ (Coordination) │ │ (Concurrency) │ │
|
||
│ └────────────────┘ └────────────────┘ └────────────────┘ │
|
||
│ │ │
|
||
│ ┌───────────────────────────┴───────────────────────────┐ │
|
||
│ │ Agent Workspaces │ │
|
||
│ │ │ │
|
||
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
|
||
│ │ │ Programmer │ │ Reviewer │ │ Tester │ │ │
|
||
│ │ │ (Opus) │ │ (Sonnet) │ │ (Sonnet) │ │ │
|
||
│ │ │ │ │ │ │ │ │ │
|
||
│ │ │ • Tools │ │ • Tools │ │ • Tools │ │ │
|
||
│ │ │ • Memory │ │ • Memory │ │ • Memory │ │ │
|
||
│ │ │ • Workspace │ │ • Workspace │ │ • Workspace │ │ │
|
||
│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │
|
||
│ │ │ │
|
||
│ └────────────────────────────────────────────────────────┘ │
|
||
│ │
|
||
└─────────────────────────────────────────────────────────────────────┘
|
||
│
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────┐
|
||
│ LLM Provider (ZAI SDK) │
|
||
└─────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## 🚀 Quick Start
|
||
|
||
### Prerequisites
|
||
|
||
```bash
|
||
# Install dependencies
|
||
bun add z-ai-web-dev-sdk
|
||
```
|
||
|
||
### Basic Usage
|
||
|
||
```typescript
|
||
import {
|
||
PipelineOrchestrator,
|
||
runWorkflow,
|
||
ContextCompactor,
|
||
TokenCounter
|
||
} from './pipeline-system';
|
||
|
||
// Initialize orchestrator
|
||
const orchestrator = new PipelineOrchestrator();
|
||
await orchestrator.initialize();
|
||
|
||
// Create a code pipeline
|
||
const pipelineId = await orchestrator.createPipeline({
|
||
name: 'Multi-Project Pipeline',
|
||
projects: [
|
||
{
|
||
id: 'project-1',
|
||
name: 'Authentication Module',
|
||
tasks: [
|
||
{ type: 'implement', description: 'Create auth module', role: 'programmer' },
|
||
{ type: 'review', description: 'Review code', role: 'reviewer' },
|
||
{ type: 'test', description: 'Test implementation', role: 'tester' }
|
||
]
|
||
}
|
||
],
|
||
roles: ['programmer', 'reviewer', 'tester'],
|
||
maxConcurrency: 12
|
||
});
|
||
|
||
// Or run a predefined workflow
|
||
const workflowId = await runWorkflow('code-pipeline', {
|
||
projectId: 'my-project'
|
||
});
|
||
```
|
||
|
||
---
|
||
|
||
## 📦 Component Overview
|
||
|
||
### Agent System (`agent-system/`)
|
||
|
||
```
|
||
agent-system/
|
||
├── core/
|
||
│ ├── orchestrator.ts # Agent lifecycle management
|
||
│ ├── token-counter.ts # Token counting & budgeting
|
||
│ ├── context-manager.ts # Context compaction logic
|
||
│ ├── subagent-spawner.ts # Subagent creation & management
|
||
│ └── summarizer.ts # LLM-powered summarization
|
||
├── agents/
|
||
│ ├── base-agent.ts # Base agent class
|
||
│ └── task-agent.ts # Task-specific agent
|
||
├── storage/
|
||
│ └── memory-store.ts # Persistent storage
|
||
└── index.ts # Main exports
|
||
```
|
||
|
||
### Pipeline System (`pipeline-system/`)
|
||
|
||
```
|
||
pipeline-system/
|
||
├── core/
|
||
│ └── state-machine.ts # Deterministic state machine
|
||
├── engine/
|
||
│ └── parallel-executor.ts # Parallel execution engine
|
||
├── events/
|
||
│ └── event-bus.ts # Event-driven coordination
|
||
├── workspace/
|
||
│ └── agent-workspace.ts # Isolated agent workspaces
|
||
├── workflows/
|
||
│ └── yaml-workflow.ts # YAML workflow parser
|
||
├── integrations/
|
||
│ └── claude-code.ts # Claude Code integration
|
||
└── index.ts # Main exports
|
||
```
|
||
|
||
---
|
||
|
||
## 💡 Usage Examples
|
||
|
||
### Token Counting
|
||
|
||
```typescript
|
||
import { TokenCounter } from './agent-system';
|
||
|
||
const counter = new TokenCounter(128000);
|
||
|
||
// Count tokens in text
|
||
const result = counter.countText("Hello, world!");
|
||
console.log(result); // { tokens: 3, characters: 13, words: 2 }
|
||
|
||
// Count conversation
|
||
const conversation = [
|
||
{ role: 'user', content: 'Hello!' },
|
||
{ role: 'assistant', content: 'Hi there!' }
|
||
];
|
||
const budget = counter.getBudget(counter.countConversation(conversation).total);
|
||
console.log(budget);
|
||
// { used: 15, remaining: 123985, total: 124000, percentageUsed: 0.01 }
|
||
```
|
||
|
||
### Context Compaction
|
||
|
||
```typescript
|
||
import { ContextCompactor } from './agent-system';
|
||
|
||
const compactor = new ContextCompactor({
|
||
maxTokens: 120000,
|
||
strategy: 'hybrid',
|
||
preserveRecentCount: 6
|
||
});
|
||
|
||
// Check if compaction needed
|
||
if (compactor.needsCompaction(messages)) {
|
||
const result = await compactor.compact(messages);
|
||
console.log(`Saved ${result.tokensSaved} tokens`);
|
||
console.log(`Compression: ${(result.compressionRatio * 100).toFixed(1)}%`);
|
||
}
|
||
```
|
||
|
||
### State Machine
|
||
|
||
```typescript
|
||
import { DeterministicStateMachine } from './pipeline-system';
|
||
|
||
const definition = {
|
||
id: 'code-pipeline',
|
||
name: 'Code Pipeline',
|
||
initial: 'start',
|
||
states: {
|
||
start: { type: 'start', onExit: [{ event: 'start', target: 'code' }] },
|
||
code: {
|
||
type: 'action',
|
||
agent: 'programmer',
|
||
onExit: [
|
||
{ event: 'completed', target: 'review' },
|
||
{ event: 'failed', target: 'failed' }
|
||
]
|
||
},
|
||
review: {
|
||
type: 'choice',
|
||
onExit: [
|
||
{ event: 'approved', target: 'end', condition: { type: 'equals', field: 'approved', value: true } },
|
||
{ event: 'rejected', target: 'code' }
|
||
]
|
||
},
|
||
end: { type: 'end' },
|
||
failed: { type: 'end' }
|
||
}
|
||
};
|
||
|
||
const sm = new DeterministicStateMachine(definition);
|
||
sm.start();
|
||
sm.sendEvent({ type: 'start', source: 'user', payload: {} });
|
||
```
|
||
|
||
### Parallel Execution
|
||
|
||
```typescript
|
||
import { ParallelExecutionEngine } from './pipeline-system';
|
||
|
||
const executor = new ParallelExecutionEngine({
|
||
maxWorkers: 4,
|
||
maxConcurrentPerWorker: 3
|
||
});
|
||
|
||
executor.start();
|
||
|
||
// Submit parallel tasks
|
||
const tasks = executor.submitBatch([
|
||
{ projectId: 'p1', role: 'programmer', type: 'implement', description: 'Auth', priority: 'high' },
|
||
{ projectId: 'p2', role: 'programmer', type: 'implement', description: 'Payment', priority: 'high' },
|
||
{ projectId: 'p3', role: 'programmer', type: 'implement', description: 'Dashboard', priority: 'medium' }
|
||
]);
|
||
```
|
||
|
||
### Event Bus
|
||
|
||
```typescript
|
||
import { EventBus, PipelineEventTypes } from './pipeline-system';
|
||
|
||
const eventBus = new EventBus();
|
||
eventBus.start();
|
||
|
||
// Subscribe to events
|
||
eventBus.subscribe({
|
||
eventType: PipelineEventTypes.AGENT_COMPLETED,
|
||
handler: async (event) => {
|
||
console.log('Agent completed:', event.payload);
|
||
// Trigger next step
|
||
eventBus.publish({
|
||
type: PipelineEventTypes.TASK_STARTED,
|
||
source: 'orchestrator',
|
||
payload: { nextAgent: 'reviewer' }
|
||
});
|
||
}
|
||
});
|
||
```
|
||
|
||
---
|
||
|
||
## 📚 API Reference
|
||
|
||
### PipelineOrchestrator
|
||
|
||
```typescript
|
||
class PipelineOrchestrator {
|
||
// Initialize the system
|
||
async initialize(): Promise<void>
|
||
|
||
// Create a pipeline
|
||
async createPipeline(config: PipelineConfig): Promise<string>
|
||
|
||
// Create from YAML workflow
|
||
async createPipelineFromYAML(workflowId: string, context?: object): Promise<string>
|
||
|
||
// Get pipeline status
|
||
getPipelineStatus(pipelineId: string): PipelineResult | undefined
|
||
|
||
// Cancel pipeline
|
||
async cancelPipeline(pipelineId: string): Promise<void>
|
||
|
||
// Subscribe to events
|
||
onEvent(eventType: string, handler: Function): () => void
|
||
|
||
// Get statistics
|
||
getStats(): object
|
||
|
||
// Shutdown
|
||
async shutdown(): Promise<void>
|
||
}
|
||
```
|
||
|
||
### Quick Start Functions
|
||
|
||
```typescript
|
||
// Create simple code pipeline
|
||
createCodePipeline(projects: ProjectConfig[]): Promise<string>
|
||
|
||
// Create parallel pipeline
|
||
createParallelPipeline(config: PipelineConfig): Promise<string>
|
||
|
||
// Run predefined workflow
|
||
runWorkflow(workflowId: string, context?: object): Promise<string>
|
||
```
|
||
|
||
---
|
||
|
||
## 🔗 Integration
|
||
|
||
### With Claude Code
|
||
|
||
```typescript
|
||
import { PipelineOrchestrator } from './pipeline-system';
|
||
|
||
const orchestrator = new PipelineOrchestrator();
|
||
await orchestrator.initialize();
|
||
|
||
// Use in Claude Code environment
|
||
const pipelineId = await orchestrator.createPipeline({
|
||
name: 'Claude Code Pipeline',
|
||
projects: [
|
||
{
|
||
id: 'claude-project',
|
||
name: 'Claude Integration',
|
||
tasks: [
|
||
{ type: 'implement', description: 'Add MCP server', role: 'programmer' },
|
||
{ type: 'review', description: 'Review changes', role: 'reviewer' },
|
||
{ type: 'test', description: 'Test integration', role: 'tester' }
|
||
]
|
||
}
|
||
]
|
||
});
|
||
```
|
||
|
||
### With OpenClaw
|
||
|
||
```typescript
|
||
import { runWorkflow, WorkflowRegistry } from './pipeline-system';
|
||
|
||
// Register custom workflow
|
||
const registry = new WorkflowRegistry();
|
||
registry.register({
|
||
id: 'custom-openclaw-workflow',
|
||
name: 'Custom Workflow',
|
||
initial: 'start',
|
||
states: { /* ... */ }
|
||
});
|
||
|
||
// Run workflow
|
||
await runWorkflow('custom-openclaw-workflow', {
|
||
projectId: 'my-project'
|
||
});
|
||
```
|
||
|
||
### Lobster-Compatible YAML
|
||
|
||
```yaml
|
||
id: code-pipeline
|
||
name: Code Pipeline
|
||
initial: start
|
||
states:
|
||
start:
|
||
type: start
|
||
on:
|
||
start: code
|
||
|
||
code:
|
||
type: action
|
||
role: programmer
|
||
timeout: 30m
|
||
retry:
|
||
maxAttempts: 2
|
||
backoff: exponential
|
||
on:
|
||
completed: review
|
||
failed: failed
|
||
|
||
review:
|
||
type: choice
|
||
on:
|
||
approved: test
|
||
rejected: code
|
||
|
||
test:
|
||
type: action
|
||
role: tester
|
||
on:
|
||
passed: end
|
||
failed: failed
|
||
|
||
end:
|
||
type: end
|
||
|
||
failed:
|
||
type: end
|
||
```
|
||
|
||
---
|
||
|
||
## 📥 Download
|
||
|
||
Pre-built packages available:
|
||
|
||
| Package | Size | Contents |
|
||
|---------|------|----------|
|
||
| `complete-agent-pipeline-system.zip` | 60KB | Both systems + docs |
|
||
| `agent-system.zip` | 27KB | Context & memory management |
|
||
| `pipeline-system.zip` | 29KB | Deterministic orchestration |
|
||
|
||
---
|
||
|
||
## 📋 Predefined Workflows
|
||
|
||
| Workflow ID | Description |
|
||
|-------------|-------------|
|
||
| `code-pipeline` | Code → Review → Test (max 3 review iterations) |
|
||
| `parallel-projects` | Run multiple projects in parallel |
|
||
| `human-approval` | Workflow with human approval gates |
|
||
|
||
---
|
||
|
||
## 🎯 Key Principles
|
||
|
||
1. **Deterministic Flow**: State machines control the pipeline, not LLM decisions
|
||
2. **Event-Driven**: Agents communicate through events, enabling loose coupling
|
||
3. **Parallel Execution**: Multiple agents work concurrently with resource isolation
|
||
4. **Workspace Isolation**: Each agent has its own tools, memory, and file space
|
||
5. **YAML Workflows**: Define pipelines declaratively, compatible with Lobster
|
||
|
||
---
|
||
|
||
## 📄 License
|
||
|
||
MIT License - Free to use, modify, and distribute.
|
||
|
||
---
|
||
|
||
## 🤝 Contributing
|
||
|
||
Contributions welcome! This system is designed for easy integration with:
|
||
- Claude Code
|
||
- OpenClaw
|
||
- Lobster
|
||
- Custom agent frameworks
|
||
|
||
---
|
||
|
||
## 📊 Project Statistics
|
||
|
||
- **Total Files**: 32 source files
|
||
- **Total Code**: ~100KB of TypeScript
|
||
- **Components**: 6 major modules
|
||
- **Predefined Workflows**: 3
|
||
|
||
---
|
||
|
||
Built with ❤️ for the AI agent community.
|