Agentic-Compaction-and-Pipleline-by-GLM-5/README.md

🤖 Agentic Compaction & Pipeline System

Complete open-source implementation of Claude Code-level architecture for deterministic multi-agent orchestration

A production-ready TypeScript implementation featuring:

• Context Compaction - Intelligent conversation summarization and token management
• Deterministic Orchestration - State machine controls flow, not LLM decisions
• Parallel Execution - Up to 12 concurrent agent sessions
• Event-Driven Coordination - Agents finish work → next step triggers automatically

---

📋 Table of Contents

• Features
• Architecture
• Quick Start
• Component Overview
• Usage Examples
• API Reference
• Integration
• Download

---

✨ Features

1. Context Compaction System

Feature	Description
Token Counting	Character-based approximation (~4 chars/token)
Conversation Summarization	LLM-powered intelligent summarization
Context Compaction	4 strategies: sliding-window, summarize-old, priority-retention, hybrid
Budget Management	Track and manage token budgets

2. Deterministic Pipeline System

Feature	Description
State Machine	Deterministic flow control (no LLM decisions)
Parallel Execution	4 projects × 3 roles = 12 concurrent sessions
Event Bus	Pub/sub coordination between agents
Workspace Isolation	Per-agent tools, memory, identity, files
YAML Workflows	OpenClaw/Lobster-compatible definitions

---

🏗️ Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                        Pipeline Orchestrator                         │
│                                                                      │
│  ┌────────────────┐  ┌────────────────┐  ┌────────────────┐         │
│  │  State Machine │  │   Event Bus    │  │ Parallel Exec  │         │
│  │ (Deterministic)│  │ (Coordination) │  │ (Concurrency)  │         │
│  └────────────────┘  └────────────────┘  └────────────────┘         │
│                              │                                       │
│  ┌───────────────────────────┴───────────────────────────┐          │
│  │                    Agent Workspaces                    │          │
│  │                                                        │          │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐    │          │
│  │  │  Programmer │  │   Reviewer  │  │    Tester   │    │          │
│  │  │   (Opus)    │  │  (Sonnet)   │  │  (Sonnet)   │    │          │
│  │  │             │  │             │  │             │    │          │
│  │  │ • Tools     │  │ • Tools     │  │ • Tools     │    │          │
│  │  │ • Memory    │  │ • Memory    │  │ • Memory    │    │          │
│  │  │ • Workspace │  │ • Workspace │  │ • Workspace │    │          │
│  │  └─────────────┘  └─────────────┘  └─────────────┘    │          │
│  │                                                        │          │
│  └────────────────────────────────────────────────────────┘          │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────────────┐
│                       LLM Provider (ZAI SDK)                         │
└─────────────────────────────────────────────────────────────────────┘

---

🚀 Quick Start

Prerequisites

# Install dependencies
bun add z-ai-web-dev-sdk

Basic Usage

import { 
  PipelineOrchestrator, 
  runWorkflow,
  ContextCompactor,
  TokenCounter 
} from './pipeline-system';

// Initialize orchestrator
const orchestrator = new PipelineOrchestrator();
await orchestrator.initialize();

// Create a code pipeline
const pipelineId = await orchestrator.createPipeline({
  name: 'Multi-Project Pipeline',
  projects: [
    {
      id: 'project-1',
      name: 'Authentication Module',
      tasks: [
        { type: 'implement', description: 'Create auth module', role: 'programmer' },
        { type: 'review', description: 'Review code', role: 'reviewer' },
        { type: 'test', description: 'Test implementation', role: 'tester' }
      ]
    }
  ],
  roles: ['programmer', 'reviewer', 'tester'],
  maxConcurrency: 12
});

// Or run a predefined workflow
const workflowId = await runWorkflow('code-pipeline', {
  projectId: 'my-project'
});

---

📦 Component Overview

Agent System (agent-system/)

agent-system/
├── core/
│   ├── orchestrator.ts       # Agent lifecycle management
│   ├── token-counter.ts      # Token counting & budgeting
│   ├── context-manager.ts    # Context compaction logic
│   ├── subagent-spawner.ts   # Subagent creation & management
│   └── summarizer.ts         # LLM-powered summarization
├── agents/
│   ├── base-agent.ts         # Base agent class
│   └── task-agent.ts         # Task-specific agent
├── storage/
│   └── memory-store.ts       # Persistent storage
└── index.ts                  # Main exports

Pipeline System (pipeline-system/)

pipeline-system/
├── core/
│   └── state-machine.ts      # Deterministic state machine
├── engine/
│   └── parallel-executor.ts  # Parallel execution engine
├── events/
│   └── event-bus.ts          # Event-driven coordination
├── workspace/
│   └── agent-workspace.ts    # Isolated agent workspaces
├── workflows/
│   └── yaml-workflow.ts      # YAML workflow parser
├── integrations/
│   └── claude-code.ts        # Claude Code integration
└── index.ts                  # Main exports

---

💡 Usage Examples

Token Counting

import { TokenCounter } from './agent-system';

const counter = new TokenCounter(128000);

// Count tokens in text
const result = counter.countText("Hello, world!");
console.log(result); // { tokens: 3, characters: 13, words: 2 }

// Count conversation
const conversation = [
  { role: 'user', content: 'Hello!' },
  { role: 'assistant', content: 'Hi there!' }
];
const budget = counter.getBudget(counter.countConversation(conversation).total);
console.log(budget);
// { used: 15, remaining: 123985, total: 124000, percentageUsed: 0.01 }

Context Compaction

import { ContextCompactor } from './agent-system';

const compactor = new ContextCompactor({
  maxTokens: 120000,
  strategy: 'hybrid',
  preserveRecentCount: 6
});

// Check if compaction needed
if (compactor.needsCompaction(messages)) {
  const result = await compactor.compact(messages);
  console.log(`Saved ${result.tokensSaved} tokens`);
  console.log(`Compression: ${(result.compressionRatio * 100).toFixed(1)}%`);
}

State Machine

import { DeterministicStateMachine } from './pipeline-system';

const definition = {
  id: 'code-pipeline',
  name: 'Code Pipeline',
  initial: 'start',
  states: {
    start: { type: 'start', onExit: [{ event: 'start', target: 'code' }] },
    code: { 
      type: 'action', 
      agent: 'programmer',
      onExit: [
        { event: 'completed', target: 'review' },
        { event: 'failed', target: 'failed' }
      ]
    },
    review: {
      type: 'choice',
      onExit: [
        { event: 'approved', target: 'end', condition: { type: 'equals', field: 'approved', value: true } },
        { event: 'rejected', target: 'code' }
      ]
    },
    end: { type: 'end' },
    failed: { type: 'end' }
  }
};

const sm = new DeterministicStateMachine(definition);
sm.start();
sm.sendEvent({ type: 'start', source: 'user', payload: {} });

Parallel Execution

import { ParallelExecutionEngine } from './pipeline-system';

const executor = new ParallelExecutionEngine({
  maxWorkers: 4,
  maxConcurrentPerWorker: 3
});

executor.start();

// Submit parallel tasks
const tasks = executor.submitBatch([
  { projectId: 'p1', role: 'programmer', type: 'implement', description: 'Auth', priority: 'high' },
  { projectId: 'p2', role: 'programmer', type: 'implement', description: 'Payment', priority: 'high' },
  { projectId: 'p3', role: 'programmer', type: 'implement', description: 'Dashboard', priority: 'medium' }
]);

Event Bus

import { EventBus, PipelineEventTypes } from './pipeline-system';

const eventBus = new EventBus();
eventBus.start();

// Subscribe to events
eventBus.subscribe({
  eventType: PipelineEventTypes.AGENT_COMPLETED,
  handler: async (event) => {
    console.log('Agent completed:', event.payload);
    // Trigger next step
    eventBus.publish({
      type: PipelineEventTypes.TASK_STARTED,
      source: 'orchestrator',
      payload: { nextAgent: 'reviewer' }
    });
  }
});

---

📚 API Reference

PipelineOrchestrator

class PipelineOrchestrator {
  // Initialize the system
  async initialize(): Promise<void>
  
  // Create a pipeline
  async createPipeline(config: PipelineConfig): Promise<string>
  
  // Create from YAML workflow
  async createPipelineFromYAML(workflowId: string, context?: object): Promise<string>
  
  // Get pipeline status
  getPipelineStatus(pipelineId: string): PipelineResult | undefined
  
  // Cancel pipeline
  async cancelPipeline(pipelineId: string): Promise<void>
  
  // Subscribe to events
  onEvent(eventType: string, handler: Function): () => void
  
  // Get statistics
  getStats(): object
  
  // Shutdown
  async shutdown(): Promise<void>
}

Quick Start Functions

// Create simple code pipeline
createCodePipeline(projects: ProjectConfig[]): Promise<string>

// Create parallel pipeline
createParallelPipeline(config: PipelineConfig): Promise<string>

// Run predefined workflow
runWorkflow(workflowId: string, context?: object): Promise<string>

---

🔗 Integration

With Claude Code

import { PipelineOrchestrator } from './pipeline-system';

const orchestrator = new PipelineOrchestrator();
await orchestrator.initialize();

// Use in Claude Code environment
const pipelineId = await orchestrator.createPipeline({
  name: 'Claude Code Pipeline',
  projects: [
    {
      id: 'claude-project',
      name: 'Claude Integration',
      tasks: [
        { type: 'implement', description: 'Add MCP server', role: 'programmer' },
        { type: 'review', description: 'Review changes', role: 'reviewer' },
        { type: 'test', description: 'Test integration', role: 'tester' }
      ]
    }
  ]
});

With OpenClaw

import { runWorkflow, WorkflowRegistry } from './pipeline-system';

// Register custom workflow
const registry = new WorkflowRegistry();
registry.register({
  id: 'custom-openclaw-workflow',
  name: 'Custom Workflow',
  initial: 'start',
  states: { /* ... */ }
});

// Run workflow
await runWorkflow('custom-openclaw-workflow', {
  projectId: 'my-project'
});

Lobster-Compatible YAML

id: code-pipeline
name: Code Pipeline
initial: start
states:
  start:
    type: start
    on:
      start: code

  code:
    type: action
    role: programmer
    timeout: 30m
    retry:
      maxAttempts: 2
      backoff: exponential
    on:
      completed: review
      failed: failed

  review:
    type: choice
    on:
      approved: test
      rejected: code

  test:
    type: action
    role: tester
    on:
      passed: end
      failed: failed

  end:
    type: end

  failed:
    type: end

---

📥 Download

Pre-built packages available:

Package	Size	Contents
complete-agent-pipeline-system.zip	60KB	Both systems + docs
agent-system.zip	27KB	Context & memory management
pipeline-system.zip	29KB	Deterministic orchestration

---

📋 Predefined Workflows

Workflow ID	Description
code-pipeline	Code → Review → Test (max 3 review iterations)
parallel-projects	Run multiple projects in parallel
human-approval	Workflow with human approval gates

---

🎯 Key Principles

1. Deterministic Flow: State machines control the pipeline, not LLM decisions
2. Event-Driven: Agents communicate through events, enabling loose coupling
3. Parallel Execution: Multiple agents work concurrently with resource isolation
4. Workspace Isolation: Each agent has its own tools, memory, and file space
5. YAML Workflows: Define pipelines declaratively, compatible with Lobster

---

📄 License

MIT License - Free to use, modify, and distribute.

---

🤝 Contributing

Contributions welcome! This system is designed for easy integration with:

• Claude Code
• OpenClaw
• Lobster
• Custom agent frameworks

---

📊 Project Statistics

• Total Files: 32 source files
• Total Code: ~100KB of TypeScript
• Components: 6 major modules
• Predefined Workflows: 3

---

Built with ❤️ for the AI agent community.