Agentic-Compaction-and-Pipleline-by-GLM-5/README.md

# Agentic Compaction & Pipeline System

<p align="center">
  <strong>A comprehensive open-source implementation of context compaction mechanisms<br>
  and deterministic multi-agent pipeline orchestration</strong>
</p>

<p align="center">
  <em>Designed for seamless integration with Claude Code, OpenClaw, and custom AI systems</em>
</p>

---

<p align="center">
  <a href="#features">Features</a> •
  <a href="#installation">Installation</a> •
  <a href="#quick-start">Quick Start</a> •
  <a href="#integrations">Integrations</a> •
  <a href="#api-reference">API Reference</a>
</p>

---

## 🤖 About This Project

<div align="center">

### ⚡ This project was 100% autonomously built by AI ⚡

**Z.AI GLM-5** created this entire codebase in a single session — designing the architecture, 
implementing all modules, writing comprehensive documentation, packaging the releases, 
and even pushing everything to this Git repository.

**[Learn more about Z.AI GLM-5 →](https://z.ai/subscribe?ic=R0K78RJKNW)**

*Yes, the README you're reading right now was written by the AI too! 🎉*

</div>

---

## Overview

This project provides two complementary systems:

1. **Agent System** - Context compaction, token management, and agent orchestration
2. **Pipeline System** - Deterministic state machine, parallel execution, and event-driven coordination

Built based on the architectural principles described in [How I Built a Deterministic Multi-Agent Dev Pipeline Inside OpenClaw](https://dev.to/ggondim/how-i-built-a-deterministic-multi-agent-dev-pipeline-inside-openclaw-and-contributed-a-missing-4ool).

---

## Features

### Agent System

- ✅ **Token Counting & Management** - Accurate token estimation with budget tracking
- ✅ **Context Compaction** - 4 strategies: sliding-window, summarize-old, priority-retention, hybrid
- ✅ **Conversation Summarization** - LLM-powered summarization with key points extraction
- ✅ **Agent Orchestration** - Lifecycle management, task routing, event handling
- ✅ **Subagent Spawning** - 6 predefined subagent types for task delegation
- ✅ **Persistent Storage** - File-based memory store for agent state
- ✅ **Claude Code Integration** - Full support for Claude Code CLI/IDE
- ✅ **OpenClaw Integration** - Native integration with OpenClaw workflows

### Pipeline System

- ✅ **Deterministic State Machine** - Flow control without LLM decisions
- ✅ **Parallel Execution Engine** - Worker pools with concurrent agent sessions
- ✅ **Event-Driven Coordination** - Pub/sub event bus with automatic trigger chains
- ✅ **Workspace Isolation** - Per-agent tools, memory, identity, permissions
- ✅ **YAML Workflow Parser** - Lobster-compatible workflow definitions
- ✅ **Claude Code Integration** - Ready-to-use integration layer

---

## Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                    AGENTIC PIPELINE SYSTEM                      │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌───────────────────────┐    ┌───────────────────────┐        │
│  │    AGENT SYSTEM       │    │   PIPELINE SYSTEM     │        │
│  ├───────────────────────┤    ├───────────────────────┤        │
│  │ • Token Counter       │    │ • State Machine       │        │
│  │ • Context Manager     │    │ • Parallel Executor   │        │
│  │ • Summarizer          │    │ • Event Bus           │        │
│  │ • Orchestrator        │    │ • Workspace Manager   │        │
│  │ • Subagent Spawner    │    │ • YAML Workflows      │        │
│  │ • Memory Store        │    │ • Claude Integration  │        │
│  │ ─────────────────────  │    └───────────────────────┘        │
│  │ INTEGRATIONS:          │                                    │
│  │ • Claude Code ✅       │                                    │
│  │ • OpenClaw ✅          │                                    │
│  └───────────────────────┘                                    │
│                                                                 │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │                    INTEGRATION LAYER                     │   │
│  │  Claude Code │ OpenClaw │ Lobster │ Custom Applications  │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘
```

### Data Flow

```
User Request → Pipeline Orchestrator → State Machine
                                          │
                                          ▼
                               Parallel Executor (Worker Pool)
                                          │
                        ┌─────────────────┼─────────────────┐
                        ▼                 ▼                 ▼
                   Agent 1           Agent 2           Agent N
                        │                 │                 │
                        ▼                 ▼                 ▼
                   Workspace         Workspace         Workspace
                        │                 │                 │
                        └─────────────────┼─────────────────┘
                                          ▼
                                    Event Bus
                                          │
                                          ▼
                                    Context Manager
                                          │
                                          ▼
                                    Summarizer (if needed)
                                          │
                                          ▼
                                    Response/Next State
```

---

## Installation

### Prerequisites

- Node.js 18+ or Bun
- TypeScript 5+

### From Source

```bash
# Clone the repository
git clone https://github.rommark.dev/admin/Agentic-Compaction-and-Pipleline-by-GLM-5.git
cd Agentic-Compaction-and-Pipleline-by-GLM-5

# Install dependencies
bun install

# Build (if needed)
bun run build
```

### Using Zip Packages

Download the appropriate package from the `downloads/` directory:

| Package | Description | Use Case |
|---------|-------------|----------|
| `agent-system.zip` | Context compaction & orchestration | Building custom AI agents |
| `pipeline-system.zip` | Deterministic pipelines | Multi-agent workflows |
| `complete-agent-pipeline-system.zip` | Full system | Complete integration |

---

## Quick Start

### Agent System

```typescript
import { ContextManager, TokenCounter, Summarizer } from './agent-system';

// Initialize components
const tokenCounter = new TokenCounter(128000); // 128k token budget
const summarizer = new Summarizer();
const contextManager = new ContextManager(tokenCounter, summarizer, {
  maxTokens: 100000,
  compactionStrategy: 'hybrid',
  reserveTokens: 20000
});

// Add messages
contextManager.addMessage({ role: 'user', content: 'Hello!' });
contextManager.addMessage({ role: 'assistant', content: 'Hi there!' });

// Check if compaction needed
if (contextManager.needsCompaction()) {
  const result = await contextManager.compact();
  console.log(`Compacted: ${result.messagesRemoved} messages removed`);
}
```

### Pipeline System

```typescript
import { DeterministicStateMachine, ParallelExecutionEngine, EventBus } from './pipeline-system';

// Define workflow
const workflow = `
name: code-pipeline
states:
  - name: analyze
    transitions:
      - to: implement
        event: analyzed
  - name: implement
    transitions:
      - to: test
        event: implemented
  - name: test
    transitions:
      - to: complete
        event: passed
`;

// Create pipeline
const eventBus = new EventBus();
const stateMachine = new DeterministicStateMachine(workflow);
const executor = new ParallelExecutionEngine({ maxConcurrency: 4 });

// Run pipeline
await stateMachine.start();
```

---

## Integrations

### Claude Code Integration

Full integration with Claude Code CLI and IDE extensions:

```typescript
import { ClaudeCodeIntegration } from './agent-system/integrations/claude-code';

// Initialize with Claude Code defaults
const claude = new ClaudeCodeIntegration({
  maxContextTokens: 200000,       // Claude's context window
  reserveTokens: 40000,           // Reserve for response
  compactionStrategy: 'hybrid',
  autoCompact: true,
  compactionThreshold: 0.8,
  enableSubagents: true,
  maxSubagents: 6,
  persistentMemory: true
});

// Add messages with automatic compaction
claude.addMessage({ role: 'user', content: 'Analyze this codebase...' });

// Get context for Claude API
const { messages, systemPrompt } = claude.getContextForAPI();

// Spawn subagents for complex tasks
const result = await claude.spawnSubagent({
  type: 'researcher',
  prompt: 'Research authentication patterns',
  priority: 'high'
});

// Parallel subagent execution (4 projects × 3 roles pattern)
const results = await claude.executeParallelSubagents([
  { type: 'explorer', prompt: 'Find security issues in frontend' },
  { type: 'explorer', prompt: 'Find security issues in backend' },
  { type: 'reviewer', prompt: 'Review API endpoints' }
]);

// Memory management
await claude.remember('userPreference', { theme: 'dark' });
const pref = await claude.recall('userPreference');

// Save/restore context
await claude.saveContext('milestone-1');
await claude.loadContext('milestone-1');

// Monitor session
const stats = claude.getTokenStats();
console.log(`Using ${stats.percentage}% of context (${stats.used}/${stats.total} tokens)`);
```

### OpenClaw Integration

Native integration with OpenClaw's deterministic multi-agent architecture:

```typescript
import { OpenClawIntegration } from './agent-system/integrations/openclaw';

// Initialize with OpenClaw-compatible config
const openclaw = new OpenClawIntegration({
  maxContextTokens: 200000,
  compactionStrategy: 'hybrid',
  workspaceIsolation: true,
  enableLobsterWorkflows: true,
  enableParallelExecution: true,
  maxParallelAgents: 12,  // 4 projects × 3 roles
  hooks: {
    onCompactionStart: (ctx) => console.log('Compacting...'),
    onCompactionEnd: (result) => console.log(`Saved ${result.tokensSaved} tokens`),
    onStateTransition: (from, to, ctx) => console.log(`${from} → ${to}`)
  }
});

// Add messages with OpenClaw context
openclaw.addMessage({
  role: 'user',
  content: 'Implement user authentication',
  tags: ['feature', 'auth'],
  references: {
    files: ['src/auth.ts', 'src/middleware.ts']
  }
});

// Spawn agents for parallel execution
const agents = await openclaw.executeParallelAgents([
  { type: 'planner', prompt: 'Plan auth architecture' },
  { type: 'researcher', prompt: 'Research JWT best practices' },
  { type: 'explorer', prompt: 'Find existing auth patterns' }
]);

// Create deterministic pipeline
const pipeline = openclaw.createPipeline({
  name: 'feature-development',
  description: 'Complete feature development workflow',
  states: [
    {
      name: 'analyze',
      type: 'parallel',
      agents: ['explorer', 'researcher'],
      transitions: [
        { target: 'design', event: 'analysis_complete' }
      ]
    },
    {
      name: 'design',
      type: 'sequential',
      agents: ['planner'],
      transitions: [
        { target: 'implement', event: 'design_approved' }
      ]
    },
    {
      name: 'implement',
      type: 'parallel',
      agents: ['coder'],
      transitions: [
        { target: 'review', event: 'implementation_complete' }
      ]
    },
    {
      name: 'review',
      type: 'sequential',
      agents: ['reviewer'],
      transitions: [
        { target: 'complete', event: 'approved' },
        { target: 'implement', event: 'rejected' }
      ]
    },
    {
      name: 'complete',
      type: 'sequential',
      transitions: []
    }
  ]
});

// Execute pipeline
await openclaw.startPipeline(pipeline.id);
await openclaw.transitionPipeline(pipeline.id, 'analysis_complete');
await openclaw.transitionPipeline(pipeline.id, 'design_approved');
// ... continue transitions

// Create isolated workspaces
const workspace = await openclaw.createWorkspace({
  permissions: ['read', 'write'],
  quota: { maxFiles: 1000, maxSize: 100 * 1024 * 1024 }
});
```

### Custom Integration

Build your own integration:

```typescript
import { 
  ContextManager, 
  TokenCounter, 
  Summarizer,
  EventBus,
  DeterministicStateMachine,
  ParallelExecutionEngine 
} from './agent-system';

class CustomAISystem {
  private contextManager: ContextManager;
  private eventBus: EventBus;
  private executor: ParallelExecutionEngine;
  
  constructor(config: any) {
    const tokenCounter = new TokenCounter(config.maxTokens);
    const summarizer = new Summarizer();
    
    this.contextManager = new ContextManager(
      tokenCounter, 
      summarizer, 
      config.compaction
    );
    
    this.eventBus = new EventBus();
    this.executor = new ParallelExecutionEngine(config.parallel);
    
    this.setupEventHandlers();
  }
  
  private setupEventHandlers() {
    this.eventBus.subscribe('context:full', async () => {
      await this.contextManager.compact();
    });
  }
  
  async process(input: string) {
    this.contextManager.addMessage({
      role: 'user',
      content: input
    });
    
    // Your custom processing logic
  }
}
```

---

## API Reference

### Agent System API

| Class | Method | Description |
|-------|--------|-------------|
| `TokenCounter` | `countTokens(text)` | Estimate token count |
| | `getRemainingBudget()` | Get remaining tokens |
| | `addUsage(count)` | Track token usage |
| `ContextManager` | `addMessage(message)` | Add message to context |
| | `needsCompaction()` | Check if compaction needed |
| | `compact()` | Perform context compaction |
| | `getActiveContext()` | Get current context |
| `Summarizer` | `summarize(messages, options)` | Generate summary |
| `Orchestrator` | `registerAgent(type, config)` | Register agent |
| | `routeTask(task)` | Route to appropriate agent |
| | `getAgentStatus(id)` | Check agent status |
| `SubagentSpawner` | `spawn(type, options)` | Create subagent |
| | `getSubagentTypes()` | List available types |
| `ClaudeCodeIntegration` | `addMessage(message)` | Add message with auto-compact |
| | `spawnSubagent(task)` | Spawn Claude Code subagent |
| | `saveContext(name)` | Persist context |
| `OpenClawIntegration` | `createPipeline(definition)` | Create OpenClaw pipeline |
| | `executeParallelAgents(tasks)` | Execute 4×3 pattern |
| | `createWorkspace(options)` | Isolated workspace |

### Pipeline System API

| Class | Method | Description |
|-------|--------|-------------|
| `DeterministicStateMachine` | `start(context)` | Start state machine |
| | `transition(event, payload)` | Trigger transition |
| | `getState()` | Get current state |
| | `canTransition(event)` | Check valid transition |
| `ParallelExecutionEngine` | `executeAll(tasks)` | Execute tasks in parallel |
| | `submitTask(task)` | Add to queue |
| | `startWorkers(count)` | Start worker threads |
| `EventBus` | `subscribe(pattern, handler)` | Subscribe to events |
| | `publish(event, data)` | Publish event |
| | `getHistory(filter)` | Get event history |
| `WorkspaceManager` | `createWorkspace(id, options)` | Create workspace |
| | `getWorkspace(id)` | Access workspace |
| | `destroyWorkspace(id)` | Cleanup workspace |
| `YAMLWorkflow` | `parse(yaml)` | Parse workflow definition |
| | `validate()` | Validate workflow |
| | `toStateMachine()` | Convert to state machine |

---

## Examples

### Example 1: Multi-Project Analysis (OpenClaw Pattern)

```typescript
import { OpenClawIntegration } from './agent-system/integrations/openclaw';

const openclaw = new OpenClawIntegration({
  maxParallelAgents: 12  // 4 projects × 3 roles
});

const projects = ['frontend', 'backend', 'mobile', 'docs'];
const roles = ['security', 'performance', 'quality'] as const;

const tasks = projects.flatMap(project =>
  roles.map(role => ({
    type: 'explorer' as const,
    prompt: `Analyze ${project} for ${role} issues`,
    context: { project, role }
  }))
);

const results = await openclaw.executeParallelAgents(tasks);

// Aggregate results by project
for (const [agentId, result] of results) {
  console.log(`Agent ${agentId}:`, result.output);
}
```

### Example 2: Context-Aware Chat with Claude Code

```typescript
import { ClaudeCodeIntegration } from './agent-system/integrations/claude-code';

class ContextAwareChat {
  private claude: ClaudeCodeIntegration;
  
  constructor() {
    this.claude = new ClaudeCodeIntegration({
      maxContextTokens: 200000,
      compactionStrategy: 'hybrid',
      priorityKeywords: ['important', 'remember', 'decision', 'error'],
      autoCompact: true,
      compactionThreshold: 0.75
    });
  }
  
  async chat(userMessage: string): Promise<string> {
    // Add user message (auto-compacts if needed)
    this.claude.addMessage({ role: 'user', content: userMessage });
    
    // Get optimized context for API
    const { messages, systemPrompt } = this.claude.getContextForAPI();
    
    // ... call Claude API with messages ...
    const response = await this.callClaudeAPI(messages, systemPrompt);
    
    // Add response to context
    this.claude.addMessage({ role: 'assistant', content: response });
    
    return response;
  }
  
  private async callClaudeAPI(messages: any[], systemPrompt?: string): Promise<string> {
    // Your Claude API implementation
    return "Response from Claude...";
  }
}
```

### Example 3: Human-in-the-Loop Workflow

```typescript
import { OpenClawIntegration } from './agent-system/integrations/openclaw';

const openclaw = new OpenClawIntegration();

const pipeline = openclaw.createPipeline({
  name: 'human-approval-workflow',
  states: [
    {
      name: 'draft',
      type: 'sequential',
      agents: ['coder'],
      transitions: [{ target: 'review', event: 'drafted' }]
    },
    {
      name: 'review',
      type: 'human-approval',
      agents: ['reviewer'],
      timeout: 86400000,  // 24 hours
      transitions: [
        { target: 'publish', event: 'approved' },
        { target: 'draft', event: 'rejected' }
      ]
    },
    {
      name: 'publish',
      type: 'sequential',
      agents: ['executor'],
      transitions: []
    }
  ]
});

await openclaw.startPipeline(pipeline.id);
```

---

## Project Structure

```
├── agent-system/                 # Context compaction system
│   ├── core/
│   │   ├── token-counter.ts      # Token counting
│   │   ├── summarizer.ts         # LLM summarization
│   │   ├── context-manager.ts    # Context compaction
│   │   ├── orchestrator.ts       # Agent orchestration
│   │   └── subagent-spawner.ts   # Subagent creation
│   ├── agents/
│   │   ├── base-agent.ts         # Base agent class
│   │   └── task-agent.ts         # Task-specific agent
│   ├── integrations/
│   │   ├── claude-code.ts        # Claude Code integration ✅
│   │   └── openclaw.ts           # OpenClaw integration ✅
│   ├── storage/
│   │   └── memory-store.ts       # Persistent storage
│   ├── utils/
│   │   └── helpers.ts            # Utility functions
│   └── index.ts                  # Main exports
│
├── pipeline-system/              # Deterministic pipeline system
│   ├── core/
│   │   └── state-machine.ts      # State machine
│   ├── engine/
│   │   └── parallel-executor.ts  # Parallel execution
│   ├── events/
│   │   └── event-bus.ts          # Event coordination
│   ├── workspace/
│   │   └── agent-workspace.ts    # Workspace isolation
│   ├── workflows/
│   │   └── yaml-workflow.ts      # YAML parser
│   ├── integrations/
│   │   └── claude-code.ts        # Claude Code integration
│   └── index.ts                  # Main exports
│
├── downloads/                    # Zip packages
│   ├── agent-system.zip
│   ├── pipeline-system.zip
│   └── complete-agent-pipeline-system.zip
│
└── README.md                     # This file
```

---

## Compaction Strategies

### 1. Sliding Window

Keeps the most recent N messages:

```typescript
const contextManager = new ContextManager(tokenCounter, summarizer, {
  compactionStrategy: 'sliding-window',
  slidingWindowSize: 50  // Keep last 50 messages
});
```

### 2. Summarize Old

Summarizes older messages into a compact summary:

```typescript
const contextManager = new ContextManager(tokenCounter, summarizer, {
  compactionStrategy: 'summarize-old',
  preserveRecentCount: 10  // Keep last 10 messages verbatim
});
```

### 3. Priority Retention

Keeps messages containing priority keywords:

```typescript
const contextManager = new ContextManager(tokenCounter, summarizer, {
  compactionStrategy: 'priority-retention',
  priorityKeywords: ['error', 'important', 'decision', 'critical']
});
```

### 4. Hybrid (Recommended)

Combines all strategies for optimal results:

```typescript
const contextManager = new ContextManager(tokenCounter, summarizer, {
  compactionStrategy: 'hybrid',
  slidingWindowSize: 30,
  preserveRecentCount: 10,
  priorityKeywords: ['error', 'important', 'decision']
});
```

---

## Contributing

Contributions are welcome! Please read our contributing guidelines before submitting PRs.

1. Fork the repository
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

---

## License

MIT License - see [LICENSE](LICENSE) for details.

---

## Acknowledgments

- Inspired by [OpenClaw](https://github.com/ggondim/openclaw) and [Lobster](https://github.com/ggondim/lobster)
- Architectural patterns from [How I Built a Deterministic Multi-Agent Dev Pipeline](https://dev.to/ggondim/how-i-built-a-deterministic-multi-agent-dev-pipeline-inside-openclaw-and-contributed-a-missing-4ool)
- Claude Code integration patterns from Anthropic's documentation

---

## Support

For issues and feature requests, please use the [GitHub Issues](https://github.rommark.dev/admin/Agentic-Compaction-and-Pipleline-by-GLM-5/issues) page.

---

<div align="center">

**Built with ❤️ by [Z.AI GLM-5](https://z.ai/subscribe?ic=R0K78RJKNW)**

*100% Autonomous AI Development*

</div>