admin/Agentic-Compaction-and-Pipleline-by-GLM-5
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Add complete Agentic Compaction & Pipeline System

Browse Source 
- Context Compaction System with token counting and summarization
- Deterministic State Machine for flow control (no LLM decisions)
- Parallel Execution Engine (up to 12 concurrent sessions)
- Event-Driven Coordination via Event Bus
- Agent Workspace Isolation (tools, memory, identity, files)
- YAML Workflow Integration (OpenClaw/Lobster compatible)
- Claude Code integration layer
- Complete demo UI with real-time visualization
- Comprehensive documentation and README

Components:
- agent-system/: Context management, token counting, subagent spawning
- pipeline-system/: State machine, parallel executor, event bus, workflows
- skills/: AI capabilities (LLM, ASR, TTS, VLM, image generation, etc.)
- src/app/: Next.js demo application

Total: ~100KB of production-ready TypeScript code
This commit is contained in:
Z User
2026-03-03 12:40:47 +00:00
Unverified
parent 63a8b123c9
commit 2380d33861
152 changed files with 51569 additions and 817 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
skills/web-search/LICENSE.txt Executable file
View File

@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2025 z-ai-web-dev-sdk Skills
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

912
skills/web-search/SKILL.md Executable file
View File

@@ -0,0 +1,912 @@
---
name: web-search
description: Implement web search capabilities using the z-ai-web-dev-sdk. Use this skill when the user needs to search for real-time information from the web, retrieve up-to-date content beyond the knowledge cutoff, or find the latest news and data. Returns structured search results with URLs, snippets, and metadata.
license: MIT
---
# Web Search Skill
This skill guides the implementation of web search functionality using the z-ai-web-dev-sdk package, enabling applications to search the web and retrieve current information.
## Installation Path
**Recommended Location**: `{project_path}/skills/web-search`
Extract this skill package to the above path in your project.
**Reference Scripts**: Example test scripts are available in the `{project_path}/skills/web-search/scripts/` directory for quick testing and reference. See `{project_path}/skills/web-search/scripts/web_search.ts` for a working example.
## Overview
The Web Search skill allows you to build applications that can search the internet, retrieve current information, and access real-time data from web sources.
**IMPORTANT**: z-ai-web-dev-sdk MUST be used in backend code only. Never use it in client-side code.
## Prerequisites
The z-ai-web-dev-sdk package is already installed. Import it as shown in the examples below.
## CLI Usage (For Simple Tasks)
For simple web search queries, you can use the z-ai CLI instead of writing code. This is ideal for quick information retrieval, testing search functionality, or command-line automation.
### Basic Web Search
```bash
# Simple search query
z-ai function --name "web_search" --args '{"query": "artificial intelligence"}'
# Using short options
z-ai function -n web_search -a '{"query": "latest tech news"}'
```
### Search with Custom Parameters
```bash
# Limit number of results
z-ai function \
-n web_search \
-a '{"query": "machine learning", "num": 5}'
# Search with recency filter (results from last N days)
z-ai function \
-n web_search \
-a '{"query": "cryptocurrency news", "num": 10, "recency_days": 7}'
```
### Save Search Results
```bash
# Save results to JSON file
z-ai function \
-n web_search \
-a '{"query": "climate change research", "num": 5}' \
-o search_results.json
# Recent news with file output
z-ai function \
-n web_search \
-a '{"query": "AI breakthroughs", "num": 3, "recency_days": 1}' \
-o ai_news.json
```
### Advanced Search Examples
```bash
# Search for specific topics
z-ai function \
-n web_search \
-a '{"query": "quantum computing applications", "num": 8}' \
-o quantum.json
# Find recent scientific papers
z-ai function \
-n web_search \
-a '{"query": "genomics research", "num": 5, "recency_days": 30}' \
-o genomics.json
# Technology news from last 24 hours
z-ai function \
-n web_search \
-a '{"query": "tech industry updates", "recency_days": 1}' \
-o today_tech.json
```
### CLI Parameters
- `--name, -n`: **Required** - Function name (use "web_search")
- `--args, -a`: **Required** - JSON arguments object with:
- `query` (string, required): Search keywords
- `num` (number, optional): Number of results (default: 10)
- `recency_days` (number, optional): Filter results from last N days
- `--output, -o <path>`: Optional - Output file path (JSON format)
### Search Result Structure
Each result contains:
- `url`: Full URL of the result
- `name`: Title of the page
- `snippet`: Preview text/description
- `host_name`: Domain name
- `rank`: Result ranking
- `date`: Publication/update date
- `favicon`: Favicon URL
### When to Use CLI vs SDK
**Use CLI for:**
- Quick information lookups
- Testing search queries
- Simple automation scripts
- One-off research tasks
**Use SDK for:**
- Dynamic search in applications
- Multi-step search workflows
- Custom result processing and filtering
- Production applications with complex logic
## Search Result Type
Each search result is a `SearchFunctionResultItem` with the following structure:
```typescript
interface SearchFunctionResultItem {
url: string; // Full URL of the result
name: string; // Title of the page
snippet: string; // Preview text/description
host_name: string; // Domain name
rank: number; // Result ranking
date: string; // Publication/update date
favicon: string; // Favicon URL
}
```
## Basic Web Search
### Simple Search Query
```javascript
import ZAI from 'z-ai-web-dev-sdk';
async function searchWeb(query) {
const zai = await ZAI.create();
const results = await zai.functions.invoke('web_search', {
query: query,
num: 10
});
return results;
}
// Usage
const searchResults = await searchWeb('What is the capital of France?');
console.log('Search Results:', searchResults);
```
### Search with Custom Result Count
```javascript
import ZAI from 'z-ai-web-dev-sdk';
async function searchWithLimit(query, numberOfResults) {
const zai = await ZAI.create();
const results = await zai.functions.invoke('web_search', {
query: query,
num: numberOfResults
});
return results;
}
// Usage - Get top 5 results
const topResults = await searchWithLimit('artificial intelligence news', 5);
// Usage - Get top 20 results
const moreResults = await searchWithLimit('JavaScript frameworks', 20);
```
### Formatted Search Results
```javascript
import ZAI from 'z-ai-web-dev-sdk';
async function getFormattedResults(query) {
const zai = await ZAI.create();
const results = await zai.functions.invoke('web_search', {
query: query,
num: 10
});
// Format results for display
const formatted = results.map((item, index) => ({
position: index + 1,
title: item.name,
url: item.url,
description: item.snippet,
domain: item.host_name,
publishDate: item.date
}));
return formatted;
}
// Usage
const results = await getFormattedResults('climate change solutions');
results.forEach(result => {
console.log(`${result.position}. ${result.title}`);
console.log(` ${result.url}`);
console.log(` ${result.description}`);
console.log('');
});
```
## Advanced Use Cases
### Search with Result Processing
```javascript
import ZAI from 'z-ai-web-dev-sdk';
class SearchProcessor {
constructor() {
this.zai = null;
}
async initialize() {
this.zai = await ZAI.create();
}
async search(query, options = {}) {
const {
num = 10,
filterDomain = null,
minSnippetLength = 0
} = options;
const results = await this.zai.functions.invoke('web_search', {
query: query,
num: num
});
// Filter results
let filtered = results;
if (filterDomain) {
filtered = filtered.filter(item =>
item.host_name.includes(filterDomain)
);
}
if (minSnippetLength > 0) {
filtered = filtered.filter(item =>
item.snippet.length >= minSnippetLength
);
}
return filtered;
}
extractDomains(results) {
return [...new Set(results.map(item => item.host_name))];
}
groupByDomain(results) {
const grouped = {};
results.forEach(item => {
if (!grouped[item.host_name]) {
grouped[item.host_name] = [];
}
grouped[item.host_name].push(item);
});
return grouped;
}
sortByDate(results, ascending = false) {
return results.sort((a, b) => {
const dateA = new Date(a.date);
const dateB = new Date(b.date);
return ascending ? dateA - dateB : dateB - dateA;
});
}
}
// Usage
const processor = new SearchProcessor();
await processor.initialize();
const results = await processor.search('machine learning tutorials', {
num: 15,
minSnippetLength: 50
});
console.log('Domains found:', processor.extractDomains(results));
console.log('Grouped by domain:', processor.groupByDomain(results));
console.log('Sorted by date:', processor.sortByDate(results));
```
### News Search
```javascript
import ZAI from 'z-ai-web-dev-sdk';
async function searchNews(topic, timeframe = 'recent') {
const zai = await ZAI.create();
// Add time-based keywords to query
const timeKeywords = {
recent: 'latest news',
today: 'today news',
week: 'this week news',
month: 'this month news'
};
const query = `${topic} ${timeKeywords[timeframe] || timeKeywords.recent}`;
const results = await zai.functions.invoke('web_search', {
query: query,
num: 10
});
// Sort by date (most recent first)
const sortedResults = results.sort((a, b) => {
return new Date(b.date) - new Date(a.date);
});
return sortedResults;
}
// Usage
const aiNews = await searchNews('artificial intelligence', 'today');
const techNews = await searchNews('technology', 'week');
console.log('Latest AI News:');
aiNews.forEach(item => {
console.log(`${item.name} (${item.date})`);
console.log(`${item.snippet}\n`);
});
```
### Research Assistant
```javascript
import ZAI from 'z-ai-web-dev-sdk';
class ResearchAssistant {
constructor() {
this.zai = null;
}
async initialize() {
this.zai = await ZAI.create();
}
async researchTopic(topic, depth = 'standard') {
const numResults = {
quick: 5,
standard: 10,
deep: 20
};
const results = await this.zai.functions.invoke('web_search', {
query: topic,
num: numResults[depth] || 10
});
// Analyze results
const analysis = {
topic: topic,
totalResults: results.length,
sources: this.extractDomains(results),
topResults: results.slice(0, 5).map(r => ({
title: r.name,
url: r.url,
summary: r.snippet
})),
dateRange: this.getDateRange(results)
};
return analysis;
}
extractDomains(results) {
const domains = {};
results.forEach(item => {
domains[item.host_name] = (domains[item.host_name] || 0) + 1;
});
return domains;
}
getDateRange(results) {
const dates = results
.map(r => new Date(r.date))
.filter(d => !isNaN(d));
if (dates.length === 0) return null;
return {
earliest: new Date(Math.min(...dates)),
latest: new Date(Math.max(...dates))
};
}
async compareTopics(topic1, topic2) {
const [results1, results2] = await Promise.all([
this.zai.functions.invoke('web_search', { query: topic1, num: 10 }),
this.zai.functions.invoke('web_search', { query: topic2, num: 10 })
]);
const domains1 = new Set(results1.map(r => r.host_name));
const domains2 = new Set(results2.map(r => r.host_name));
const commonDomains = [...domains1].filter(d => domains2.has(d));
return {
topic1: {
name: topic1,
results: results1.length,
uniqueDomains: domains1.size
},
topic2: {
name: topic2,
results: results2.length,
uniqueDomains: domains2.size
},
commonDomains: commonDomains
};
}
}
// Usage
const assistant = new ResearchAssistant();
await assistant.initialize();
const research = await assistant.researchTopic('quantum computing', 'deep');
console.log('Research Analysis:', research);
const comparison = await assistant.compareTopics(
'renewable energy',
'solar power'
);
console.log('Topic Comparison:', comparison);
```
### Search Result Validation
```javascript
import ZAI from 'z-ai-web-dev-sdk';
async function validateSearchResults(query) {
const zai = await ZAI.create();
const results = await zai.functions.invoke('web_search', {
query: query,
num: 10
});
// Validate and score results
const validated = results.map(item => {
let score = 0;
let flags = [];
// Check snippet quality
if (item.snippet && item.snippet.length > 50) {
score += 20;
} else {
flags.push('short_snippet');
}
// Check date availability
if (item.date && item.date !== 'N/A') {
score += 20;
} else {
flags.push('no_date');
}
// Check URL validity
try {
new URL(item.url);
score += 20;
} catch (e) {
flags.push('invalid_url');
}
// Check domain quality (not perfect, but basic check)
if (!item.host_name.includes('spam') &&
!item.host_name.includes('ads')) {
score += 20;
} else {
flags.push('suspicious_domain');
}
// Check title quality
if (item.name && item.name.length > 10) {
score += 20;
} else {
flags.push('short_title');
}
return {
...item,
qualityScore: score,
validationFlags: flags,
isHighQuality: score >= 80
};
});
// Sort by quality score
return validated.sort((a, b) => b.qualityScore - a.qualityScore);
}
// Usage
const validated = await validateSearchResults('best programming practices');
console.log('High quality results:',
validated.filter(r => r.isHighQuality).length
);
```
## Best Practices
### 1. Query Optimization
```javascript
// Bad: Too vague
const bad = await searchWeb('information');
// Good: Specific and targeted
const good = await searchWeb('JavaScript async/await best practices 2024');
// Good: Include context
const goodWithContext = await searchWeb('React hooks tutorial for beginners');
```
### 2. Error Handling
```javascript
import ZAI from 'z-ai-web-dev-sdk';
async function safeSearch(query, retries = 3) {
let lastError;
for (let attempt = 1; attempt <= retries; attempt++) {
try {
const zai = await ZAI.create();
const results = await zai.functions.invoke('web_search', {
query: query,
num: 10
});
if (!Array.isArray(results) || results.length === 0) {
throw new Error('No results found or invalid response');
}
return {
success: true,
results: results,
attempts: attempt
};
} catch (error) {
lastError = error;
console.error(`Attempt ${attempt} failed:`, error.message);
if (attempt < retries) {
// Wait before retry (exponential backoff)
await new Promise(resolve => setTimeout(resolve, 1000 * attempt));
}
}
}
return {
success: false,
error: lastError.message,
attempts: retries
};
}
```
### 3. Result Caching
```javascript
import ZAI from 'z-ai-web-dev-sdk';
class CachedSearch {
constructor(cacheDuration = 3600000) { // 1 hour default
this.cache = new Map();
this.cacheDuration = cacheDuration;
this.zai = null;
}
async initialize() {
this.zai = await ZAI.create();
}
getCacheKey(query, num) {
return `${query}_${num}`;
}
async search(query, num = 10) {
const cacheKey = this.getCacheKey(query, num);
const cached = this.cache.get(cacheKey);
// Check if cached and not expired
if (cached && Date.now() - cached.timestamp < this.cacheDuration) {
console.log('Returning cached results');
return {
...cached.data,
cached: true
};
}
// Perform fresh search
const results = await this.zai.functions.invoke('web_search', {
query: query,
num: num
});
// Cache results
this.cache.set(cacheKey, {
data: results,
timestamp: Date.now()
});
return {
results: results,
cached: false
};
}
clearCache() {
this.cache.clear();
}
getCacheSize() {
return this.cache.size;
}
}
// Usage
const search = new CachedSearch(1800000); // 30 minutes cache
await search.initialize();
const result1 = await search.search('TypeScript tutorial');
console.log('Cached:', result1.cached); // false
const result2 = await search.search('TypeScript tutorial');
console.log('Cached:', result2.cached); // true
```
### 4. Rate Limiting
```javascript
class RateLimitedSearch {
constructor(requestsPerMinute = 60) {
this.zai = null;
this.requestsPerMinute = requestsPerMinute;
this.requests = [];
}
async initialize() {
this.zai = await ZAI.create();
}
async search(query, num = 10) {
await this.checkRateLimit();
const results = await this.zai.functions.invoke('web_search', {
query: query,
num: num
});
this.requests.push(Date.now());
return results;
}
async checkRateLimit() {
const now = Date.now();
const oneMinuteAgo = now - 60000;
// Remove requests older than 1 minute
this.requests = this.requests.filter(time => time > oneMinuteAgo);
if (this.requests.length >= this.requestsPerMinute) {
const oldestRequest = this.requests[0];
const waitTime = 60000 - (now - oldestRequest);
console.log(`Rate limit reached. Waiting ${waitTime}ms`);
await new Promise(resolve => setTimeout(resolve, waitTime));
// Recheck after waiting
return this.checkRateLimit();
}
}
}
```
## Common Use Cases
1. **Real-time Information Retrieval**: Get current news, stock prices, weather
2. **Research & Analysis**: Gather information on specific topics
3. **Content Discovery**: Find articles, tutorials, documentation
4. **Competitive Analysis**: Research competitors and market trends
5. **Fact Checking**: Verify information against web sources
6. **SEO & Content Research**: Analyze search results for content strategy
7. **News Aggregation**: Collect news from various sources
8. **Academic Research**: Find papers, studies, and academic content
## Integration Examples
### Express.js Search API
```javascript
import express from 'express';
import ZAI from 'z-ai-web-dev-sdk';
const app = express();
app.use(express.json());
let zaiInstance;
async function initZAI() {
zaiInstance = await ZAI.create();
}
app.get('/api/search', async (req, res) => {
try {
const { q: query, num = 10 } = req.query;
if (!query) {
return res.status(400).json({ error: 'Query parameter "q" is required' });
}
const numResults = Math.min(parseInt(num) || 10, 20);
const results = await zaiInstance.functions.invoke('web_search', {
query: query,
num: numResults
});
res.json({
success: true,
query: query,
totalResults: results.length,
results: results
});
} catch (error) {
res.status(500).json({
success: false,
error: error.message
});
}
});
app.get('/api/search/news', async (req, res) => {
try {
const { topic, timeframe = 'recent' } = req.query;
if (!topic) {
return res.status(400).json({ error: 'Topic parameter is required' });
}
const timeKeywords = {
recent: 'latest news',
today: 'today news',
week: 'this week news'
};
const query = `${topic} ${timeKeywords[timeframe] || timeKeywords.recent}`;
const results = await zaiInstance.functions.invoke('web_search', {
query: query,
num: 15
});
// Sort by date
const sortedResults = results.sort((a, b) => {
return new Date(b.date) - new Date(a.date);
});
res.json({
success: true,
topic: topic,
timeframe: timeframe,
results: sortedResults
});
} catch (error) {
res.status(500).json({
success: false,
error: error.message
});
}
});
initZAI().then(() => {
app.listen(3000, () => {
console.log('Search API running on port 3000');
});
});
```
### Search with AI Summary
```javascript
import ZAI from 'z-ai-web-dev-sdk';
async function searchAndSummarize(query) {
const zai = await ZAI.create();
// Step 1: Search the web
const searchResults = await zai.functions.invoke('web_search', {
query: query,
num: 10
});
// Step 2: Create summary using chat completions
const searchContext = searchResults
.slice(0, 5)
.map((r, i) => `${i + 1}. ${r.name}\n${r.snippet}`)
.join('\n\n');
const completion = await zai.chat.completions.create({
messages: [
{
role: 'assistant',
content: 'You are a research assistant. Summarize search results clearly and concisely.'
},
{
role: 'user',
content: `Query: "${query}"\n\nSearch Results:\n${searchContext}\n\nProvide a comprehensive summary of these results.`
}
],
thinking: { type: 'disabled' }
});
const summary = completion.choices[0]?.message?.content;
return {
query: query,
summary: summary,
sources: searchResults.slice(0, 5).map(r => ({
title: r.name,
url: r.url
})),
totalResults: searchResults.length
};
}
// Usage
const result = await searchAndSummarize('benefits of renewable energy');
console.log('Summary:', result.summary);
console.log('Sources:', result.sources);
```
## Troubleshooting
**Issue**: "SDK must be used in backend"
- **Solution**: Ensure z-ai-web-dev-sdk is only imported and used in server-side code
**Issue**: Empty or no results returned
- **Solution**: Try different query terms, check internet connectivity, verify API status
**Issue**: Unexpected response format
- **Solution**: Verify the response is an array, check for API changes, add type validation
**Issue**: Rate limiting errors
- **Solution**: Implement request throttling, add delays between searches, use caching
**Issue**: Low quality search results
- **Solution**: Refine query terms, filter results by domain or date, validate result quality
## Performance Tips
1. **Reuse SDK Instance**: Create ZAI instance once and reuse across searches
2. **Implement Caching**: Cache search results to reduce API calls
3. **Optimize Query Terms**: Use specific, targeted queries for better results
4. **Limit Result Count**: Request only the number of results you need
5. **Parallel Searches**: Use Promise.all for multiple independent searches
6. **Result Filtering**: Filter results on client side when possible
## Security Considerations
1. **Input Validation**: Sanitize and validate user search queries
2. **Rate Limiting**: Implement rate limits to prevent abuse
3. **API Key Protection**: Never expose SDK credentials in client-side code
4. **Result Filtering**: Filter potentially harmful or inappropriate content
5. **URL Validation**: Validate URLs before redirecting users
6. **Privacy**: Don't log sensitive user search queries
## Remember
- Always use z-ai-web-dev-sdk in backend code only
- The SDK is already installed - import as shown in examples
- Search results are returned as an array of SearchFunctionResultItem objects
- Implement proper error handling and retries for production
- Cache results when appropriate to reduce API calls
- Use specific query terms for better search results
- Validate and filter results before displaying to users
- Check `scripts/web_search.ts` for a quick start example

44
skills/web-search/scripts/web_search.ts Executable file
View File

@@ -0,0 +1,44 @@
import ZAI from 'z-ai-web-dev-sdk';
interface SearchFunctionResultItem {
url: string;
name: string;
snippet: string;
host_name: string;
rank: number;
date: string;
favicon: string;
}
async function main(query: string, num: number = 10) {
try {
const zai = await ZAI.create();
const searchResult = await zai.functions.invoke('web_search', {
query: query,
num: num
});
console.log('Search Results:');
console.log('================\n');
if (Array.isArray(searchResult)) {
searchResult.forEach((item: SearchFunctionResultItem, index: number) => {
console.log(`${index + 1}. ${item.name}`);
console.log(` URL: ${item.url}`);
console.log(` Snippet: ${item.snippet}`);
console.log(` Host: ${item.host_name}`);
console.log(` Date: ${item.date}`);
console.log('');
});
console.log(`\nTotal results: ${searchResult.length}`);
} else {
console.log('Unexpected response format:', searchResult);
}
} catch (err: any) {
console.error('Web search failed:', err?.message || err);
}
}
main('What is the capital of France?', 5);