Agentic-Compaction-and-Pipleline-by-GLM-5/README.md

Agentic Compaction & Pipeline System

A comprehensive open-source implementation of context compaction mechanisms
and deterministic multi-agent pipeline orchestration

Designed for seamless integration with Claude Code, OpenClaw, and custom AI systems

---

Features • Installation • Quick Start • Integrations • API Reference

---

🤖 About This Project

⚡ This project was 100% autonomously built by AI ⚡

Z.AI GLM-5 created this entire codebase in a single session — designing the architecture, implementing all modules, writing comprehensive documentation, packaging the releases, and even pushing everything to this Git repository.

Learn more about Z.AI GLM-5 →

Yes, the README you're reading right now was written by the AI too! 🎉

---

Overview

This project provides two complementary systems:

1. Agent System - Context compaction, token management, and agent orchestration
2. Pipeline System - Deterministic state machine, parallel execution, and event-driven coordination

Built based on the architectural principles described in How I Built a Deterministic Multi-Agent Dev Pipeline Inside OpenClaw.

---

Features

Agent System

• ✅ Token Counting & Management - Accurate token estimation with budget tracking
• ✅ Context Compaction - 4 strategies: sliding-window, summarize-old, priority-retention, hybrid
• ✅ Conversation Summarization - LLM-powered summarization with key points extraction
• ✅ Agent Orchestration - Lifecycle management, task routing, event handling
• ✅ Subagent Spawning - 6 predefined subagent types for task delegation
• ✅ Persistent Storage - File-based memory store for agent state
• ✅ Claude Code Integration - Full support for Claude Code CLI/IDE
• ✅ OpenClaw Integration - Native integration with OpenClaw workflows

Pipeline System

• ✅ Deterministic State Machine - Flow control without LLM decisions
• ✅ Parallel Execution Engine - Worker pools with concurrent agent sessions
• ✅ Event-Driven Coordination - Pub/sub event bus with automatic trigger chains
• ✅ Workspace Isolation - Per-agent tools, memory, identity, permissions
• ✅ YAML Workflow Parser - Lobster-compatible workflow definitions
• ✅ Claude Code Integration - Ready-to-use integration layer

---

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    AGENTIC PIPELINE SYSTEM                      │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌───────────────────────┐    ┌───────────────────────┐        │
│  │    AGENT SYSTEM       │    │   PIPELINE SYSTEM     │        │
│  ├───────────────────────┤    ├───────────────────────┤        │
│  │ • Token Counter       │    │ • State Machine       │        │
│  │ • Context Manager     │    │ • Parallel Executor   │        │
│  │ • Summarizer          │    │ • Event Bus           │        │
│  │ • Orchestrator        │    │ • Workspace Manager   │        │
│  │ • Subagent Spawner    │    │ • YAML Workflows      │        │
│  │ • Memory Store        │    │ • Claude Integration  │        │
│  │ ─────────────────────  │    └───────────────────────┘        │
│  │ INTEGRATIONS:          │                                    │
│  │ • Claude Code ✅       │                                    │
│  │ • OpenClaw ✅          │                                    │
│  └───────────────────────┘                                    │
│                                                                 │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │                    INTEGRATION LAYER                     │   │
│  │  Claude Code │ OpenClaw │ Lobster │ Custom Applications  │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Data Flow

User Request → Pipeline Orchestrator → State Machine
                                          │
                                          ▼
                               Parallel Executor (Worker Pool)
                                          │
                        ┌─────────────────┼─────────────────┐
                        ▼                 ▼                 ▼
                   Agent 1           Agent 2           Agent N
                        │                 │                 │
                        ▼                 ▼                 ▼
                   Workspace         Workspace         Workspace
                        │                 │                 │
                        └─────────────────┼─────────────────┘
                                          ▼
                                    Event Bus
                                          │
                                          ▼
                                    Context Manager
                                          │
                                          ▼
                                    Summarizer (if needed)
                                          │
                                          ▼
                                    Response/Next State

---

Installation

Prerequisites

• Node.js 18+ or Bun
• TypeScript 5+

From Source

# Clone the repository
git clone https://github.rommark.dev/admin/Agentic-Compaction-and-Pipleline-by-GLM-5.git
cd Agentic-Compaction-and-Pipleline-by-GLM-5

# Install dependencies
bun install

# Build (if needed)
bun run build

Using Zip Packages

Download the appropriate package from the downloads/ directory:

Package	Description	Use Case
agent-system.zip	Context compaction & orchestration	Building custom AI agents
pipeline-system.zip	Deterministic pipelines	Multi-agent workflows
complete-agent-pipeline-system.zip	Full system	Complete integration

---

Quick Start

Agent System

import { ContextManager, TokenCounter, Summarizer } from './agent-system';

// Initialize components
const tokenCounter = new TokenCounter(128000); // 128k token budget
const summarizer = new Summarizer();
const contextManager = new ContextManager(tokenCounter, summarizer, {
  maxTokens: 100000,
  compactionStrategy: 'hybrid',
  reserveTokens: 20000
});

// Add messages
contextManager.addMessage({ role: 'user', content: 'Hello!' });
contextManager.addMessage({ role: 'assistant', content: 'Hi there!' });

// Check if compaction needed
if (contextManager.needsCompaction()) {
  const result = await contextManager.compact();
  console.log(`Compacted: ${result.messagesRemoved} messages removed`);
}

Pipeline System

import { DeterministicStateMachine, ParallelExecutionEngine, EventBus } from './pipeline-system';

// Define workflow
const workflow = `
name: code-pipeline
states:
  - name: analyze
    transitions:
      - to: implement
        event: analyzed
  - name: implement
    transitions:
      - to: test
        event: implemented
  - name: test
    transitions:
      - to: complete
        event: passed
`;

// Create pipeline
const eventBus = new EventBus();
const stateMachine = new DeterministicStateMachine(workflow);
const executor = new ParallelExecutionEngine({ maxConcurrency: 4 });

// Run pipeline
await stateMachine.start();

---

Integrations

Claude Code Integration

Full integration with Claude Code CLI and IDE extensions:

import { ClaudeCodeIntegration } from './agent-system/integrations/claude-code';

// Initialize with Claude Code defaults
const claude = new ClaudeCodeIntegration({
  maxContextTokens: 200000,       // Claude's context window
  reserveTokens: 40000,           // Reserve for response
  compactionStrategy: 'hybrid',
  autoCompact: true,
  compactionThreshold: 0.8,
  enableSubagents: true,
  maxSubagents: 6,
  persistentMemory: true
});

// Add messages with automatic compaction
claude.addMessage({ role: 'user', content: 'Analyze this codebase...' });

// Get context for Claude API
const { messages, systemPrompt } = claude.getContextForAPI();

// Spawn subagents for complex tasks
const result = await claude.spawnSubagent({
  type: 'researcher',
  prompt: 'Research authentication patterns',
  priority: 'high'
});

// Parallel subagent execution (4 projects × 3 roles pattern)
const results = await claude.executeParallelSubagents([
  { type: 'explorer', prompt: 'Find security issues in frontend' },
  { type: 'explorer', prompt: 'Find security issues in backend' },
  { type: 'reviewer', prompt: 'Review API endpoints' }
]);

// Memory management
await claude.remember('userPreference', { theme: 'dark' });
const pref = await claude.recall('userPreference');

// Save/restore context
await claude.saveContext('milestone-1');
await claude.loadContext('milestone-1');

// Monitor session
const stats = claude.getTokenStats();
console.log(`Using ${stats.percentage}% of context (${stats.used}/${stats.total} tokens)`);

OpenClaw Integration

Native integration with OpenClaw's deterministic multi-agent architecture:

import { OpenClawIntegration } from './agent-system/integrations/openclaw';

// Initialize with OpenClaw-compatible config
const openclaw = new OpenClawIntegration({
  maxContextTokens: 200000,
  compactionStrategy: 'hybrid',
  workspaceIsolation: true,
  enableLobsterWorkflows: true,
  enableParallelExecution: true,
  maxParallelAgents: 12,  // 4 projects × 3 roles
  hooks: {
    onCompactionStart: (ctx) => console.log('Compacting...'),
    onCompactionEnd: (result) => console.log(`Saved ${result.tokensSaved} tokens`),
    onStateTransition: (from, to, ctx) => console.log(`${from} → ${to}`)
  }
});

// Add messages with OpenClaw context
openclaw.addMessage({
  role: 'user',
  content: 'Implement user authentication',
  tags: ['feature', 'auth'],
  references: {
    files: ['src/auth.ts', 'src/middleware.ts']
  }
});

// Spawn agents for parallel execution
const agents = await openclaw.executeParallelAgents([
  { type: 'planner', prompt: 'Plan auth architecture' },
  { type: 'researcher', prompt: 'Research JWT best practices' },
  { type: 'explorer', prompt: 'Find existing auth patterns' }
]);

// Create deterministic pipeline
const pipeline = openclaw.createPipeline({
  name: 'feature-development',
  description: 'Complete feature development workflow',
  states: [
    {
      name: 'analyze',
      type: 'parallel',
      agents: ['explorer', 'researcher'],
      transitions: [
        { target: 'design', event: 'analysis_complete' }
      ]
    },
    {
      name: 'design',
      type: 'sequential',
      agents: ['planner'],
      transitions: [
        { target: 'implement', event: 'design_approved' }
      ]
    },
    {
      name: 'implement',
      type: 'parallel',
      agents: ['coder'],
      transitions: [
        { target: 'review', event: 'implementation_complete' }
      ]
    },
    {
      name: 'review',
      type: 'sequential',
      agents: ['reviewer'],
      transitions: [
        { target: 'complete', event: 'approved' },
        { target: 'implement', event: 'rejected' }
      ]
    },
    {
      name: 'complete',
      type: 'sequential',
      transitions: []
    }
  ]
});

// Execute pipeline
await openclaw.startPipeline(pipeline.id);
await openclaw.transitionPipeline(pipeline.id, 'analysis_complete');
await openclaw.transitionPipeline(pipeline.id, 'design_approved');
// ... continue transitions

// Create isolated workspaces
const workspace = await openclaw.createWorkspace({
  permissions: ['read', 'write'],
  quota: { maxFiles: 1000, maxSize: 100 * 1024 * 1024 }
});

Custom Integration

Build your own integration:

import { 
  ContextManager, 
  TokenCounter, 
  Summarizer,
  EventBus,
  DeterministicStateMachine,
  ParallelExecutionEngine 
} from './agent-system';

class CustomAISystem {
  private contextManager: ContextManager;
  private eventBus: EventBus;
  private executor: ParallelExecutionEngine;
  
  constructor(config: any) {
    const tokenCounter = new TokenCounter(config.maxTokens);
    const summarizer = new Summarizer();
    
    this.contextManager = new ContextManager(
      tokenCounter, 
      summarizer, 
      config.compaction
    );
    
    this.eventBus = new EventBus();
    this.executor = new ParallelExecutionEngine(config.parallel);
    
    this.setupEventHandlers();
  }
  
  private setupEventHandlers() {
    this.eventBus.subscribe('context:full', async () => {
      await this.contextManager.compact();
    });
  }
  
  async process(input: string) {
    this.contextManager.addMessage({
      role: 'user',
      content: input
    });
    
    // Your custom processing logic
  }
}

---

API Reference

Agent System API

Class	Method	Description
TokenCounter	countTokens(text)	Estimate token count
	getRemainingBudget()	Get remaining tokens
	addUsage(count)	Track token usage
ContextManager	addMessage(message)	Add message to context
	needsCompaction()	Check if compaction needed
	compact()	Perform context compaction
	getActiveContext()	Get current context
Summarizer	summarize(messages, options)	Generate summary
Orchestrator	registerAgent(type, config)	Register agent
	routeTask(task)	Route to appropriate agent
	getAgentStatus(id)	Check agent status
SubagentSpawner	spawn(type, options)	Create subagent
	getSubagentTypes()	List available types
ClaudeCodeIntegration	addMessage(message)	Add message with auto-compact
	spawnSubagent(task)	Spawn Claude Code subagent
	saveContext(name)	Persist context
OpenClawIntegration	createPipeline(definition)	Create OpenClaw pipeline
	executeParallelAgents(tasks)	Execute 4×3 pattern
	createWorkspace(options)	Isolated workspace

Pipeline System API

Class	Method	Description
DeterministicStateMachine	start(context)	Start state machine
	transition(event, payload)	Trigger transition
	getState()	Get current state
	canTransition(event)	Check valid transition
ParallelExecutionEngine	executeAll(tasks)	Execute tasks in parallel
	submitTask(task)	Add to queue
	startWorkers(count)	Start worker threads
EventBus	subscribe(pattern, handler)	Subscribe to events
	publish(event, data)	Publish event
	getHistory(filter)	Get event history
WorkspaceManager	createWorkspace(id, options)	Create workspace
	getWorkspace(id)	Access workspace
	destroyWorkspace(id)	Cleanup workspace
YAMLWorkflow	parse(yaml)	Parse workflow definition
	validate()	Validate workflow
	toStateMachine()	Convert to state machine

---

Examples

Example 1: Multi-Project Analysis (OpenClaw Pattern)

import { OpenClawIntegration } from './agent-system/integrations/openclaw';

const openclaw = new OpenClawIntegration({
  maxParallelAgents: 12  // 4 projects × 3 roles
});

const projects = ['frontend', 'backend', 'mobile', 'docs'];
const roles = ['security', 'performance', 'quality'] as const;

const tasks = projects.flatMap(project =>
  roles.map(role => ({
    type: 'explorer' as const,
    prompt: `Analyze ${project} for ${role} issues`,
    context: { project, role }
  }))
);

const results = await openclaw.executeParallelAgents(tasks);

// Aggregate results by project
for (const [agentId, result] of results) {
  console.log(`Agent ${agentId}:`, result.output);
}

Example 2: Context-Aware Chat with Claude Code

import { ClaudeCodeIntegration } from './agent-system/integrations/claude-code';

class ContextAwareChat {
  private claude: ClaudeCodeIntegration;
  
  constructor() {
    this.claude = new ClaudeCodeIntegration({
      maxContextTokens: 200000,
      compactionStrategy: 'hybrid',
      priorityKeywords: ['important', 'remember', 'decision', 'error'],
      autoCompact: true,
      compactionThreshold: 0.75
    });
  }
  
  async chat(userMessage: string): Promise<string> {
    // Add user message (auto-compacts if needed)
    this.claude.addMessage({ role: 'user', content: userMessage });
    
    // Get optimized context for API
    const { messages, systemPrompt } = this.claude.getContextForAPI();
    
    // ... call Claude API with messages ...
    const response = await this.callClaudeAPI(messages, systemPrompt);
    
    // Add response to context
    this.claude.addMessage({ role: 'assistant', content: response });
    
    return response;
  }
  
  private async callClaudeAPI(messages: any[], systemPrompt?: string): Promise<string> {
    // Your Claude API implementation
    return "Response from Claude...";
  }
}

Example 3: Human-in-the-Loop Workflow

import { OpenClawIntegration } from './agent-system/integrations/openclaw';

const openclaw = new OpenClawIntegration();

const pipeline = openclaw.createPipeline({
  name: 'human-approval-workflow',
  states: [
    {
      name: 'draft',
      type: 'sequential',
      agents: ['coder'],
      transitions: [{ target: 'review', event: 'drafted' }]
    },
    {
      name: 'review',
      type: 'human-approval',
      agents: ['reviewer'],
      timeout: 86400000,  // 24 hours
      transitions: [
        { target: 'publish', event: 'approved' },
        { target: 'draft', event: 'rejected' }
      ]
    },
    {
      name: 'publish',
      type: 'sequential',
      agents: ['executor'],
      transitions: []
    }
  ]
});

await openclaw.startPipeline(pipeline.id);

---

Project Structure

├── agent-system/                 # Context compaction system
│   ├── core/
│   │   ├── token-counter.ts      # Token counting
│   │   ├── summarizer.ts         # LLM summarization
│   │   ├── context-manager.ts    # Context compaction
│   │   ├── orchestrator.ts       # Agent orchestration
│   │   └── subagent-spawner.ts   # Subagent creation
│   ├── agents/
│   │   ├── base-agent.ts         # Base agent class
│   │   └── task-agent.ts         # Task-specific agent
│   ├── integrations/
│   │   ├── claude-code.ts        # Claude Code integration ✅
│   │   └── openclaw.ts           # OpenClaw integration ✅
│   ├── storage/
│   │   └── memory-store.ts       # Persistent storage
│   ├── utils/
│   │   └── helpers.ts            # Utility functions
│   └── index.ts                  # Main exports
│
├── pipeline-system/              # Deterministic pipeline system
│   ├── core/
│   │   └── state-machine.ts      # State machine
│   ├── engine/
│   │   └── parallel-executor.ts  # Parallel execution
│   ├── events/
│   │   └── event-bus.ts          # Event coordination
│   ├── workspace/
│   │   └── agent-workspace.ts    # Workspace isolation
│   ├── workflows/
│   │   └── yaml-workflow.ts      # YAML parser
│   ├── integrations/
│   │   └── claude-code.ts        # Claude Code integration
│   └── index.ts                  # Main exports
│
├── downloads/                    # Zip packages
│   ├── agent-system.zip
│   ├── pipeline-system.zip
│   └── complete-agent-pipeline-system.zip
│
└── README.md                     # This file

---

Compaction Strategies

1. Sliding Window

Keeps the most recent N messages:

const contextManager = new ContextManager(tokenCounter, summarizer, {
  compactionStrategy: 'sliding-window',
  slidingWindowSize: 50  // Keep last 50 messages
});

2. Summarize Old

Summarizes older messages into a compact summary:

const contextManager = new ContextManager(tokenCounter, summarizer, {
  compactionStrategy: 'summarize-old',
  preserveRecentCount: 10  // Keep last 10 messages verbatim
});

3. Priority Retention

Keeps messages containing priority keywords:

const contextManager = new ContextManager(tokenCounter, summarizer, {
  compactionStrategy: 'priority-retention',
  priorityKeywords: ['error', 'important', 'decision', 'critical']
});

4. Hybrid (Recommended)

Combines all strategies for optimal results:

const contextManager = new ContextManager(tokenCounter, summarizer, {
  compactionStrategy: 'hybrid',
  slidingWindowSize: 30,
  preserveRecentCount: 10,
  priorityKeywords: ['error', 'important', 'decision']
});

---

Contributing

Contributions are welcome! Please read our contributing guidelines before submitting PRs.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

---

License

MIT License - see LICENSE for details.

---

Acknowledgments

• Inspired by OpenClaw and Lobster
• Architectural patterns from How I Built a Deterministic Multi-Agent Dev Pipeline
• Claude Code integration patterns from Anthropic's documentation

---

Support

For issues and feature requests, please use the GitHub Issues page.

---

Built with ❤️ by Z.AI GLM-5

100% Autonomous AI Development