feat: Add complete Agentic Compaction & Pipeline System

- Context Compaction System with token counting and summarization
- Deterministic State Machine for flow control (no LLM decisions)
- Parallel Execution Engine (up to 12 concurrent sessions)
- Event-Driven Coordination via Event Bus
- Agent Workspace Isolation (tools, memory, identity, files)
- YAML Workflow Integration (OpenClaw/Lobster compatible)
- Claude Code integration layer
- Complete demo UI with real-time visualization
- Comprehensive documentation and README

Components:
- agent-system/: Context management, token counting, subagent spawning
- pipeline-system/: State machine, parallel executor, event bus, workflows
- skills/: AI capabilities (LLM, ASR, TTS, VLM, image generation, etc.)
- src/app/: Next.js demo application

Total: ~100KB of production-ready TypeScript code
This commit is contained in:
Z User
2026-03-03 12:40:47 +00:00
Unverified
parent 63a8b123c9
commit 2380d33861
152 changed files with 51569 additions and 817 deletions

624
README.md
View File

@@ -1,141 +1,493 @@
# 🚀 Welcome to Z.ai Code Scaffold
# 🤖 Agentic Compaction & Pipeline System
A modern, production-ready web application scaffold powered by cutting-edge technologies, designed to accelerate your development with [Z.ai](https://chat.z.ai)'s AI-powered coding assistance.
> **Complete open-source implementation of Claude Code-level architecture for deterministic multi-agent orchestration**
## ✨ Technology Stack
This scaffold provides a robust foundation built with:
### 🎯 Core Framework
- **⚡ Next.js 16** - The React framework for production with App Router
- **📘 TypeScript 5** - Type-safe JavaScript for better developer experience
- **🎨 Tailwind CSS 4** - Utility-first CSS framework for rapid UI development
### 🧩 UI Components & Styling
- **🧩 shadcn/ui** - High-quality, accessible components built on Radix UI
- **🎯 Lucide React** - Beautiful & consistent icon library
- **🌈 Framer Motion** - Production-ready motion library for React
- **🎨 Next Themes** - Perfect dark mode in 2 lines of code
### 📋 Forms & Validation
- **🎣 React Hook Form** - Performant forms with easy validation
- **✅ Zod** - TypeScript-first schema validation
### 🔄 State Management & Data Fetching
- **🐻 Zustand** - Simple, scalable state management
- **🔄 TanStack Query** - Powerful data synchronization for React
- **🌐 Fetch** - Promise-based HTTP request
### 🗄️ Database & Backend
- **🗄️ Prisma** - Next-generation TypeScript ORM
- **🔐 NextAuth.js** - Complete open-source authentication solution
### 🎨 Advanced UI Features
- **📊 TanStack Table** - Headless UI for building tables and datagrids
- **🖱️ DND Kit** - Modern drag and drop toolkit for React
- **📊 Recharts** - Redefined chart library built with React and D3
- **🖼️ Sharp** - High performance image processing
### 🌍 Internationalization & Utilities
- **🌍 Next Intl** - Internationalization library for Next.js
- **📅 Date-fns** - Modern JavaScript date utility library
- **🪝 ReactUse** - Collection of essential React hooks for modern development
## 🎯 Why This Scaffold?
- **🏎️ Fast Development** - Pre-configured tooling and best practices
- **🎨 Beautiful UI** - Complete shadcn/ui component library with advanced interactions
- **🔒 Type Safety** - Full TypeScript configuration with Zod validation
- **📱 Responsive** - Mobile-first design principles with smooth animations
- **🗄️ Database Ready** - Prisma ORM configured for rapid backend development
- **🔐 Auth Included** - NextAuth.js for secure authentication flows
- **📊 Data Visualization** - Charts, tables, and drag-and-drop functionality
- **🌍 i18n Ready** - Multi-language support with Next Intl
- **🚀 Production Ready** - Optimized build and deployment settings
- **🤖 AI-Friendly** - Structured codebase perfect for AI assistance
## 🚀 Quick Start
```bash
# Install dependencies
bun install
# Start development server
bun run dev
# Build for production
bun run build
# Start production server
bun start
```
Open [http://localhost:3000](http://localhost:3000) to see your application running.
## 🤖 Powered by Z.ai
This scaffold is optimized for use with [Z.ai](https://chat.z.ai) - your AI assistant for:
- **💻 Code Generation** - Generate components, pages, and features instantly
- **🎨 UI Development** - Create beautiful interfaces with AI assistance
- **🔧 Bug Fixing** - Identify and resolve issues with intelligent suggestions
- **📝 Documentation** - Auto-generate comprehensive documentation
- **🚀 Optimization** - Performance improvements and best practices
Ready to build something amazing? Start chatting with Z.ai at [chat.z.ai](https://chat.z.ai) and experience the future of AI-powered development!
## 📁 Project Structure
```
src/
├── app/ # Next.js App Router pages
├── components/ # Reusable React components
│ └── ui/ # shadcn/ui components
├── hooks/ # Custom React hooks
└── lib/ # Utility functions and configurations
```
## 🎨 Available Features & Components
This scaffold includes a comprehensive set of modern web development tools:
### 🧩 UI Components (shadcn/ui)
- **Layout**: Card, Separator, Aspect Ratio, Resizable Panels
- **Forms**: Input, Textarea, Select, Checkbox, Radio Group, Switch
- **Feedback**: Alert, Toast (Sonner), Progress, Skeleton
- **Navigation**: Breadcrumb, Menubar, Navigation Menu, Pagination
- **Overlay**: Dialog, Sheet, Popover, Tooltip, Hover Card
- **Data Display**: Badge, Avatar, Calendar
### 📊 Advanced Data Features
- **Tables**: Powerful data tables with sorting, filtering, pagination (TanStack Table)
- **Charts**: Beautiful visualizations with Recharts
- **Forms**: Type-safe forms with React Hook Form + Zod validation
### 🎨 Interactive Features
- **Animations**: Smooth micro-interactions with Framer Motion
- **Drag & Drop**: Modern drag-and-drop functionality with DND Kit
- **Theme Switching**: Built-in dark/light mode support
### 🔐 Backend Integration
- **Authentication**: Ready-to-use auth flows with NextAuth.js
- **Database**: Type-safe database operations with Prisma
- **API Client**: HTTP requests with Fetch + TanStack Query
- **State Management**: Simple and scalable with Zustand
### 🌍 Production Features
- **Internationalization**: Multi-language support with Next Intl
- **Image Optimization**: Automatic image processing with Sharp
- **Type Safety**: End-to-end TypeScript with Zod validation
- **Essential Hooks**: 100+ useful React hooks with ReactUse for common patterns
## 🤝 Get Started with Z.ai
1. **Clone this scaffold** to jumpstart your project
2. **Visit [chat.z.ai](https://chat.z.ai)** to access your AI coding assistant
3. **Start building** with intelligent code generation and assistance
4. **Deploy with confidence** using the production-ready setup
A production-ready TypeScript implementation featuring:
- **Context Compaction** - Intelligent conversation summarization and token management
- **Deterministic Orchestration** - State machine controls flow, not LLM decisions
- **Parallel Execution** - Up to 12 concurrent agent sessions
- **Event-Driven Coordination** - Agents finish work → next step triggers automatically
---
Built with ❤️ for the developer community. Supercharged by [Z.ai](https://chat.z.ai) 🚀
## 📋 Table of Contents
- [Features](#-features)
- [Architecture](#-architecture)
- [Quick Start](#-quick-start)
- [Component Overview](#-component-overview)
- [Usage Examples](#-usage-examples)
- [API Reference](#-api-reference)
- [Integration](#-integration)
- [Download](#-download)
---
## ✨ Features
### 1. Context Compaction System
| Feature | Description |
|---------|-------------|
| Token Counting | Character-based approximation (~4 chars/token) |
| Conversation Summarization | LLM-powered intelligent summarization |
| Context Compaction | 4 strategies: sliding-window, summarize-old, priority-retention, hybrid |
| Budget Management | Track and manage token budgets |
### 2. Deterministic Pipeline System
| Feature | Description |
|---------|-------------|
| State Machine | Deterministic flow control (no LLM decisions) |
| Parallel Execution | 4 projects × 3 roles = 12 concurrent sessions |
| Event Bus | Pub/sub coordination between agents |
| Workspace Isolation | Per-agent tools, memory, identity, files |
| YAML Workflows | OpenClaw/Lobster-compatible definitions |
---
## 🏗️ Architecture
```
┌─────────────────────────────────────────────────────────────────────┐
│ Pipeline Orchestrator │
│ │
│ ┌────────────────┐ ┌────────────────┐ ┌────────────────┐ │
│ │ State Machine │ │ Event Bus │ │ Parallel Exec │ │
│ │ (Deterministic)│ │ (Coordination) │ │ (Concurrency) │ │
│ └────────────────┘ └────────────────┘ └────────────────┘ │
│ │ │
│ ┌───────────────────────────┴───────────────────────────┐ │
│ │ Agent Workspaces │ │
│ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ Programmer │ │ Reviewer │ │ Tester │ │ │
│ │ │ (Opus) │ │ (Sonnet) │ │ (Sonnet) │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ • Tools │ │ • Tools │ │ • Tools │ │ │
│ │ │ • Memory │ │ • Memory │ │ • Memory │ │ │
│ │ │ • Workspace │ │ • Workspace │ │ • Workspace │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ │ │ │
│ └────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────┐
│ LLM Provider (ZAI SDK) │
└─────────────────────────────────────────────────────────────────────┘
```
---
## 🚀 Quick Start
### Prerequisites
```bash
# Install dependencies
bun add z-ai-web-dev-sdk
```
### Basic Usage
```typescript
import {
PipelineOrchestrator,
runWorkflow,
ContextCompactor,
TokenCounter
} from './pipeline-system';
// Initialize orchestrator
const orchestrator = new PipelineOrchestrator();
await orchestrator.initialize();
// Create a code pipeline
const pipelineId = await orchestrator.createPipeline({
name: 'Multi-Project Pipeline',
projects: [
{
id: 'project-1',
name: 'Authentication Module',
tasks: [
{ type: 'implement', description: 'Create auth module', role: 'programmer' },
{ type: 'review', description: 'Review code', role: 'reviewer' },
{ type: 'test', description: 'Test implementation', role: 'tester' }
]
}
],
roles: ['programmer', 'reviewer', 'tester'],
maxConcurrency: 12
});
// Or run a predefined workflow
const workflowId = await runWorkflow('code-pipeline', {
projectId: 'my-project'
});
```
---
## 📦 Component Overview
### Agent System (`agent-system/`)
```
agent-system/
├── core/
│ ├── orchestrator.ts # Agent lifecycle management
│ ├── token-counter.ts # Token counting & budgeting
│ ├── context-manager.ts # Context compaction logic
│ ├── subagent-spawner.ts # Subagent creation & management
│ └── summarizer.ts # LLM-powered summarization
├── agents/
│ ├── base-agent.ts # Base agent class
│ └── task-agent.ts # Task-specific agent
├── storage/
│ └── memory-store.ts # Persistent storage
└── index.ts # Main exports
```
### Pipeline System (`pipeline-system/`)
```
pipeline-system/
├── core/
│ └── state-machine.ts # Deterministic state machine
├── engine/
│ └── parallel-executor.ts # Parallel execution engine
├── events/
│ └── event-bus.ts # Event-driven coordination
├── workspace/
│ └── agent-workspace.ts # Isolated agent workspaces
├── workflows/
│ └── yaml-workflow.ts # YAML workflow parser
├── integrations/
│ └── claude-code.ts # Claude Code integration
└── index.ts # Main exports
```
---
## 💡 Usage Examples
### Token Counting
```typescript
import { TokenCounter } from './agent-system';
const counter = new TokenCounter(128000);
// Count tokens in text
const result = counter.countText("Hello, world!");
console.log(result); // { tokens: 3, characters: 13, words: 2 }
// Count conversation
const conversation = [
{ role: 'user', content: 'Hello!' },
{ role: 'assistant', content: 'Hi there!' }
];
const budget = counter.getBudget(counter.countConversation(conversation).total);
console.log(budget);
// { used: 15, remaining: 123985, total: 124000, percentageUsed: 0.01 }
```
### Context Compaction
```typescript
import { ContextCompactor } from './agent-system';
const compactor = new ContextCompactor({
maxTokens: 120000,
strategy: 'hybrid',
preserveRecentCount: 6
});
// Check if compaction needed
if (compactor.needsCompaction(messages)) {
const result = await compactor.compact(messages);
console.log(`Saved ${result.tokensSaved} tokens`);
console.log(`Compression: ${(result.compressionRatio * 100).toFixed(1)}%`);
}
```
### State Machine
```typescript
import { DeterministicStateMachine } from './pipeline-system';
const definition = {
id: 'code-pipeline',
name: 'Code Pipeline',
initial: 'start',
states: {
start: { type: 'start', onExit: [{ event: 'start', target: 'code' }] },
code: {
type: 'action',
agent: 'programmer',
onExit: [
{ event: 'completed', target: 'review' },
{ event: 'failed', target: 'failed' }
]
},
review: {
type: 'choice',
onExit: [
{ event: 'approved', target: 'end', condition: { type: 'equals', field: 'approved', value: true } },
{ event: 'rejected', target: 'code' }
]
},
end: { type: 'end' },
failed: { type: 'end' }
}
};
const sm = new DeterministicStateMachine(definition);
sm.start();
sm.sendEvent({ type: 'start', source: 'user', payload: {} });
```
### Parallel Execution
```typescript
import { ParallelExecutionEngine } from './pipeline-system';
const executor = new ParallelExecutionEngine({
maxWorkers: 4,
maxConcurrentPerWorker: 3
});
executor.start();
// Submit parallel tasks
const tasks = executor.submitBatch([
{ projectId: 'p1', role: 'programmer', type: 'implement', description: 'Auth', priority: 'high' },
{ projectId: 'p2', role: 'programmer', type: 'implement', description: 'Payment', priority: 'high' },
{ projectId: 'p3', role: 'programmer', type: 'implement', description: 'Dashboard', priority: 'medium' }
]);
```
### Event Bus
```typescript
import { EventBus, PipelineEventTypes } from './pipeline-system';
const eventBus = new EventBus();
eventBus.start();
// Subscribe to events
eventBus.subscribe({
eventType: PipelineEventTypes.AGENT_COMPLETED,
handler: async (event) => {
console.log('Agent completed:', event.payload);
// Trigger next step
eventBus.publish({
type: PipelineEventTypes.TASK_STARTED,
source: 'orchestrator',
payload: { nextAgent: 'reviewer' }
});
}
});
```
---
## 📚 API Reference
### PipelineOrchestrator
```typescript
class PipelineOrchestrator {
// Initialize the system
async initialize(): Promise<void>
// Create a pipeline
async createPipeline(config: PipelineConfig): Promise<string>
// Create from YAML workflow
async createPipelineFromYAML(workflowId: string, context?: object): Promise<string>
// Get pipeline status
getPipelineStatus(pipelineId: string): PipelineResult | undefined
// Cancel pipeline
async cancelPipeline(pipelineId: string): Promise<void>
// Subscribe to events
onEvent(eventType: string, handler: Function): () => void
// Get statistics
getStats(): object
// Shutdown
async shutdown(): Promise<void>
}
```
### Quick Start Functions
```typescript
// Create simple code pipeline
createCodePipeline(projects: ProjectConfig[]): Promise<string>
// Create parallel pipeline
createParallelPipeline(config: PipelineConfig): Promise<string>
// Run predefined workflow
runWorkflow(workflowId: string, context?: object): Promise<string>
```
---
## 🔗 Integration
### With Claude Code
```typescript
import { PipelineOrchestrator } from './pipeline-system';
const orchestrator = new PipelineOrchestrator();
await orchestrator.initialize();
// Use in Claude Code environment
const pipelineId = await orchestrator.createPipeline({
name: 'Claude Code Pipeline',
projects: [
{
id: 'claude-project',
name: 'Claude Integration',
tasks: [
{ type: 'implement', description: 'Add MCP server', role: 'programmer' },
{ type: 'review', description: 'Review changes', role: 'reviewer' },
{ type: 'test', description: 'Test integration', role: 'tester' }
]
}
]
});
```
### With OpenClaw
```typescript
import { runWorkflow, WorkflowRegistry } from './pipeline-system';
// Register custom workflow
const registry = new WorkflowRegistry();
registry.register({
id: 'custom-openclaw-workflow',
name: 'Custom Workflow',
initial: 'start',
states: { /* ... */ }
});
// Run workflow
await runWorkflow('custom-openclaw-workflow', {
projectId: 'my-project'
});
```
### Lobster-Compatible YAML
```yaml
id: code-pipeline
name: Code Pipeline
initial: start
states:
start:
type: start
on:
start: code
code:
type: action
role: programmer
timeout: 30m
retry:
maxAttempts: 2
backoff: exponential
on:
completed: review
failed: failed
review:
type: choice
on:
approved: test
rejected: code
test:
type: action
role: tester
on:
passed: end
failed: failed
end:
type: end
failed:
type: end
```
---
## 📥 Download
Pre-built packages available:
| Package | Size | Contents |
|---------|------|----------|
| `complete-agent-pipeline-system.zip` | 60KB | Both systems + docs |
| `agent-system.zip` | 27KB | Context & memory management |
| `pipeline-system.zip` | 29KB | Deterministic orchestration |
---
## 📋 Predefined Workflows
| Workflow ID | Description |
|-------------|-------------|
| `code-pipeline` | Code → Review → Test (max 3 review iterations) |
| `parallel-projects` | Run multiple projects in parallel |
| `human-approval` | Workflow with human approval gates |
---
## 🎯 Key Principles
1. **Deterministic Flow**: State machines control the pipeline, not LLM decisions
2. **Event-Driven**: Agents communicate through events, enabling loose coupling
3. **Parallel Execution**: Multiple agents work concurrently with resource isolation
4. **Workspace Isolation**: Each agent has its own tools, memory, and file space
5. **YAML Workflows**: Define pipelines declaratively, compatible with Lobster
---
## 📄 License
MIT License - Free to use, modify, and distribute.
---
## 🤝 Contributing
Contributions welcome! This system is designed for easy integration with:
- Claude Code
- OpenClaw
- Lobster
- Custom agent frameworks
---
## 📊 Project Statistics
- **Total Files**: 32 source files
- **Total Code**: ~100KB of TypeScript
- **Components**: 6 major modules
- **Predefined Workflows**: 3
---
Built with ❤️ for the AI agent community.