681 lines
35 KiB
Markdown
681 lines
35 KiB
Markdown
# 🦆 GOOSE SUPER - CORE SUBSYSTEMS DETAILED SPECIFICATIONS
|
|
|
|
## Table of Contents
|
|
1. [Playwright Integration](#1-playwright-integration)
|
|
2. [Browser Use System](#2-browser-use-system)
|
|
3. [Computer Use System](#3-computer-use-system)
|
|
4. [Full Vision Capabilities](#4-full-vision-capabilities)
|
|
5. [Vibe Server Management](#5-vibe-server-management)
|
|
|
|
---
|
|
|
|
# 1. Playwright Integration
|
|
|
|
## Purpose
|
|
Automate web browsers with precision - navigate, click, fill forms, extract data, take screenshots.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌────────────────────────────────────────────────────────────────┐
|
|
│ PLAYWRIGHT SUBSYSTEM │
|
|
├────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ User: "Search for hotels in Dubai on Booking.com" │
|
|
│ ↓ │
|
|
│ ┌──────────────────────────────────────────────────────────┐ │
|
|
│ │ IQ Exchange Translation Layer │ │
|
|
│ │ → BROWSER action detected │ │
|
|
│ │ → Route to Playwright Bridge │ │
|
|
│ └──────────────────────────────────────────────────────────┘ │
|
|
│ ↓ │
|
|
│ ┌──────────────────────────────────────────────────────────┐ │
|
|
│ │ Playwright Bridge (bin/playwright-bridge.js) │ │
|
|
│ │ │ │
|
|
│ │ Commands: │ │
|
|
│ │ navigate <url> → Go to URL │ │
|
|
│ │ click <selector|text> → Click element │ │
|
|
│ │ fill <selector> <text> → Type in input │ │
|
|
│ │ type <text> → Type at cursor │ │
|
|
│ │ press <key> → Press key (Enter, Tab, etc) │ │
|
|
│ │ wait <selector> <ms> → Wait for element │ │
|
|
│ │ screenshot <file> → Capture page │ │
|
|
│ │ content → Get page text │ │
|
|
│ │ elements → List all elements │ │
|
|
│ │ close → Close browser session │ │
|
|
│ └──────────────────────────────────────────────────────────┘ │
|
|
│ ↓ │
|
|
│ ┌──────────────────────────────────────────────────────────┐ │
|
|
│ │ Chromium Browser (Headless or Visible) │ │
|
|
│ │ • Persistent session across commands │ │
|
|
│ │ • Cookies & auth remembered │ │
|
|
│ │ • DevTools access for debugging │ │
|
|
│ └──────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
└────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## Current State vs. Enhancement
|
|
|
|
| Feature | Current | Enhanced |
|
|
|---------|---------|----------|
|
|
| Session persistence | ✅ Basic | ✅ Full cookie/auth persistence |
|
|
| Element selection | Text/CSS only | Smart AI-assisted selector |
|
|
| Form filling | Manual fields | Auto-detect all form fields |
|
|
| Multi-tab | ❌ Single tab | ✅ Multiple tabs |
|
|
| Screenshots | ✅ Basic | ✅ With element annotations |
|
|
| DOM extraction | ❌ | ✅ Structured for AI analysis |
|
|
|
|
## Enhanced Playwright Bridge Commands
|
|
|
|
```javascript
|
|
// NEW: Smart element finding (AI-assisted)
|
|
node playwright-bridge.js smart-click "the search button"
|
|
node playwright-bridge.js smart-fill "email" "user@example.com"
|
|
|
|
// NEW: Element discovery for AI
|
|
node playwright-bridge.js list-interactive // Lists all clickable/fillable elements
|
|
node playwright-bridge.js describe-page // AI-friendly page description
|
|
|
|
// NEW: Multi-tab
|
|
node playwright-bridge.js new-tab
|
|
node playwright-bridge.js switch-tab 2
|
|
node playwright-bridge.js list-tabs
|
|
|
|
// NEW: Authentication helpers
|
|
node playwright-bridge.js save-cookies "site-name"
|
|
node playwright-bridge.js load-cookies "site-name"
|
|
|
|
// NEW: Form automation
|
|
node playwright-bridge.js auto-fill-form '{"email":"x","password":"y"}'
|
|
```
|
|
|
|
## Integration with IQ Exchange
|
|
|
|
```javascript
|
|
// In lib/iq-exchange.mjs - TASK_PATTERNS
|
|
browser: /\b(website|browser|search|navigate|booking\.com|amazon|google|fill.*form|click.*button|login|signup)\b/i
|
|
|
|
// Translation example:
|
|
// User: "Book a hotel in Dubai on Booking.com"
|
|
// IQ Exchange generates:
|
|
[
|
|
'node playwright-bridge.js navigate "https://booking.com"',
|
|
'node playwright-bridge.js wait "[name=ss]" 5000',
|
|
'node playwright-bridge.js fill "[name=ss]" "Dubai hotels"',
|
|
'node playwright-bridge.js click "[type=submit]"',
|
|
'node playwright-bridge.js screenshot "search-results.png"'
|
|
]
|
|
```
|
|
|
|
---
|
|
|
|
# 2. Browser Use System
|
|
|
|
## Purpose
|
|
Built-in browser panel inside Goose for:
|
|
1. **App Preview** - See your code rendered live
|
|
2. **Web Browsing** - AI can browse the web inside Goose
|
|
3. **Automation Control** - Visual feedback for Playwright actions
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌────────────────────────────────────────────────────────────────┐
|
|
│ BROWSER USE SYSTEM │
|
|
├────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ ┌──────────────────────────────────────────────────────────┐ │
|
|
│ │ BUILT-IN BROWSER (Electron BrowserView) │ │
|
|
│ │ │ │
|
|
│ │ ┌───────────────────────────────────────────────────┐ │ │
|
|
│ │ │ 🌐 https://localhost:5173 🔄 ◀ ▶ 🔍 🛠️ 📸 │ │ │
|
|
│ │ ├───────────────────────────────────────────────────┤ │ │
|
|
│ │ │ │ │ │
|
|
│ │ │ YOUR APP RENDERS HERE │ │ │
|
|
│ │ │ │ │ │
|
|
│ │ │ [Live preview of HTML/CSS/JS you write] │ │ │
|
|
│ │ │ │ │ │
|
|
│ │ │ │ │ │
|
|
│ │ └───────────────────────────────────────────────────┘ │ │
|
|
│ │ │ │
|
|
│ │ Toolbar: │ │
|
|
│ │ 🔄 Refresh ◀ Back ▶ Forward 🔍 URL Bar │ │
|
|
│ │ 🛠️ DevTools 📸 Screenshot 📍 Element Inspect │ │
|
|
│ └──────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ Modes: │
|
|
│ [Preview] - Shows your local app (default) │
|
|
│ [Browse] - Navigate to any URL │
|
|
│ [Automate]- Watch Playwright actions in real-time │
|
|
│ │
|
|
└────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## Features
|
|
|
|
### Preview Mode (Vibe Coding)
|
|
```
|
|
User: "Make me a landing page"
|
|
↓
|
|
Goose writes HTML/CSS/JS
|
|
↓
|
|
Auto-starts Vite dev server (port 5173)
|
|
↓
|
|
Built-in browser loads http://localhost:5173
|
|
↓
|
|
User: "Make the header blue"
|
|
↓
|
|
Goose edits CSS → Hot reload → Preview updates instantly
|
|
```
|
|
|
|
### Browse Mode
|
|
```
|
|
User: "Go to GitHub and star the browser-use repo"
|
|
↓
|
|
Built-in browser navigates to github.com
|
|
↓
|
|
AI uses Playwright to interact with the page
|
|
↓
|
|
User can SEE every action happening
|
|
```
|
|
|
|
### Automate Mode (Visible Automation)
|
|
```
|
|
When Playwright runs:
|
|
1. Built-in browser shows the exact page
|
|
2. Red highlight appears on clicked elements
|
|
3. Green highlight on filled inputs
|
|
4. Status bar shows current action
|
|
```
|
|
|
|
## IPC Handlers (main.cjs)
|
|
|
|
```javascript
|
|
// Browser panel control
|
|
ipcMain.handle('browser:create-view', async (event, options) => {
|
|
browserView = new BrowserView({ webPreferences });
|
|
mainWindow.addBrowserView(browserView);
|
|
return { success: true };
|
|
});
|
|
|
|
ipcMain.handle('browser:navigate', async (event, url) => {
|
|
await browserView.webContents.loadURL(url);
|
|
return { success: true, url };
|
|
});
|
|
|
|
ipcMain.handle('browser:screenshot', async () => {
|
|
const image = await browserView.webContents.capturePage();
|
|
return { success: true, base64: image.toDataURL() };
|
|
});
|
|
|
|
ipcMain.handle('browser:execute-js', async (event, script) => {
|
|
const result = await browserView.webContents.executeJavaScript(script);
|
|
return { success: true, result };
|
|
});
|
|
|
|
ipcMain.handle('browser:get-dom', async () => {
|
|
const dom = await browserView.webContents.executeJavaScript(`
|
|
Array.from(document.querySelectorAll('a, button, input, select, textarea'))
|
|
.map(el => ({
|
|
tag: el.tagName,
|
|
text: el.textContent?.trim().slice(0, 50),
|
|
id: el.id,
|
|
name: el.name,
|
|
type: el.type,
|
|
placeholder: el.placeholder
|
|
}))
|
|
`);
|
|
return { success: true, elements: dom };
|
|
});
|
|
```
|
|
|
|
---
|
|
|
|
# 3. Computer Use System
|
|
|
|
## Purpose
|
|
Control the Windows desktop - open apps, click UI elements, type text, take screenshots.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌────────────────────────────────────────────────────────────────┐
|
|
│ COMPUTER USE SYSTEM │
|
|
├────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ User: "Open Spotify and play some jazz" │
|
|
│ ↓ │
|
|
│ ┌──────────────────────────────────────────────────────────┐ │
|
|
│ │ IQ Exchange Translation Layer │ │
|
|
│ │ → DESKTOP action detected │ │
|
|
│ │ → Route to Computer Use module │ │
|
|
│ └──────────────────────────────────────────────────────────┘ │
|
|
│ ↓ │
|
|
│ ┌────────────────────────────────────┬─────────────────────┐ │
|
|
│ │ input.ps1 (PowerShell) │ computer-use.cjs │ │
|
|
│ │ (Current - 66KB!) │ (Node.js native) │ │
|
|
│ │ │ │ │
|
|
│ │ • UIAutomation via .NET │ • robotjs for speed │ │
|
|
│ │ • OCR via Windows API │ • screenshot-desktop│ │
|
|
│ │ • Window management │ • Cross-platform │ │
|
|
│ └────────────────────────────────────┴─────────────────────┘ │
|
|
│ ↓ │
|
|
│ ┌──────────────────────────────────────────────────────────┐ │
|
|
│ │ Windows Desktop │ │
|
|
│ │ • Apps, dialogs, menus │ │
|
|
│ │ • Task bar, system tray │ │
|
|
│ │ • Any visible UI element │ │
|
|
│ └──────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
└────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## Available Commands (input.ps1)
|
|
|
|
### Vision Commands (The Eyes)
|
|
```powershell
|
|
# See what's on screen
|
|
input.ps1 screenshot "screen.png" # Capture full screen
|
|
input.ps1 ocr "full" # Read ALL text on screen
|
|
input.ps1 ocr "region" 100 100 500 400 # Read text in region
|
|
input.ps1 app_state "Notepad" # Get UI tree of app (buttons, menus, etc.)
|
|
```
|
|
|
|
### Navigation Commands
|
|
```powershell
|
|
# Open and focus apps
|
|
input.ps1 open "Spotify" # Launch or focus app
|
|
input.ps1 startmenu # Open Windows Start menu
|
|
input.ps1 focus "Save As" # Focus specific dialog/element
|
|
input.ps1 waitfor "Ready" 10 # Wait for text to appear (max 10s)
|
|
```
|
|
|
|
### Interaction Commands (Smart - via UIAutomation)
|
|
```powershell
|
|
# Click by element name (RELIABLE!)
|
|
input.ps1 uiclick "Play" # Click button named "Play"
|
|
input.ps1 uiclick "File" # Click menu item
|
|
input.ps1 uipress "Remember me" # Toggle checkbox
|
|
input.ps1 uipress "Jazz" # Select list item
|
|
|
|
# Type text
|
|
input.ps1 type "Hello World" # Type into focused element
|
|
input.ps1 keyboard "search query" # Alternative typing method
|
|
input.ps1 key "Enter" # Press single key
|
|
input.ps1 hotkey "Ctrl+S" # Key combination
|
|
```
|
|
|
|
### Fallback Commands (Coordinate-based)
|
|
```powershell
|
|
# When UIAutomation fails, use coordinates
|
|
input.ps1 mouse 500 300 # Move mouse to x,y
|
|
input.ps1 click # Click at current position
|
|
input.ps1 drag 100 100 500 500 # Drag from point to point
|
|
```
|
|
|
|
## Example Flow
|
|
|
|
```
|
|
User: "Open Paint and draw a red circle"
|
|
|
|
IQ Exchange generates:
|
|
1. input.ps1 open "Paint"
|
|
2. input.ps1 waitfor "Untitled" 5
|
|
3. input.ps1 uiclick "Brushes"
|
|
4. input.ps1 uiclick "Red"
|
|
5. input.ps1 mouse 400 300
|
|
6. input.ps1 drag 400 300 500 400
|
|
|
|
After each step:
|
|
→ Screenshot
|
|
→ OCR/app_state to verify
|
|
→ If failed → Self-heal and retry
|
|
```
|
|
|
|
## Enhancements Needed
|
|
|
|
| Current | Enhanced |
|
|
|---------|----------|
|
|
| PowerShell scripts (slow startup) | Node.js native with robotjs |
|
|
| Basic element finding | Smart element grounding with labels |
|
|
| Manual verification | Auto-screenshot after each action |
|
|
| English-only OCR | Multi-language OCR support |
|
|
|
|
---
|
|
|
|
# 4. Full Vision Capabilities
|
|
|
|
## Purpose
|
|
Let the AI "see" the screen like a human - understand what's visible, where elements are, and verify actions worked.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌────────────────────────────────────────────────────────────────┐
|
|
│ VISION SYSTEM │
|
|
├────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ ┌──────────────────────────────────────────────────────────┐ │
|
|
│ │ SCREENSHOT LAYER │ │
|
|
│ │ • Full screen capture │ │
|
|
│ │ • Window-specific capture │ │
|
|
│ │ • Region capture │ │
|
|
│ │ • Before/after comparison │ │
|
|
│ └──────────────────────────────────────────────────────────┘ │
|
|
│ ↓ │
|
|
│ ┌──────────────────────────────────────────────────────────┐ │
|
|
│ │ ANALYSIS LAYER │ │
|
|
│ │ │ │
|
|
│ │ ┌────────────────┬────────────────┬──────────────────┐ │ │
|
|
│ │ │ OCR Engine │ UI Automation │ AI Vision (LLM) │ │ │
|
|
│ │ │ │ │ │ │ │
|
|
│ │ │ • Read text │ • Element tree │ • Understand │ │ │
|
|
│ │ │ • Find words │ • Button names │ context │ │ │
|
|
│ │ │ • Coordinates │ • Input fields │ • Verify success │ │ │
|
|
│ │ │ of text │ • Menus │ • Describe scene │ │ │
|
|
│ │ └────────────────┴────────────────┴──────────────────┘ │ │
|
|
│ └──────────────────────────────────────────────────────────┘ │
|
|
│ ↓ │
|
|
│ ┌──────────────────────────────────────────────────────────┐ │
|
|
│ │ GROUNDING LAYER (Element Labeling) │ │
|
|
│ │ │ │
|
|
│ │ Screenshot with overlays: │ │
|
|
│ │ ┌─────────────────────────────────────────┐ │ │
|
|
│ │ │ [1] File [2] Edit [3] View │ │ │
|
|
│ │ │ ┌─────────────────────────────┐ │ │ │
|
|
│ │ │ │ [4] Search... │ │ │ │
|
|
│ │ │ └─────────────────────────────┘ │ │ │
|
|
│ │ │ │ │ │
|
|
│ │ │ [5] Save [6] Cancel │ │ │
|
|
│ │ └─────────────────────────────────────────┘ │ │
|
|
│ │ │ │
|
|
│ │ AI can say: "Click element [5]" → We click Save button │ │
|
|
│ └──────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
└────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## Vision Flow
|
|
|
|
### Before Action
|
|
```
|
|
1. Screenshot current state
|
|
2. OCR to find text locations
|
|
3. UIAutomation to get element names
|
|
4. Create "grounding map" of clickable elements
|
|
5. Send to AI: "Here's what I see on screen..."
|
|
```
|
|
|
|
### After Action
|
|
```
|
|
1. Screenshot new state
|
|
2. Compare with previous
|
|
3. OCR to verify expected text appeared
|
|
4. Send to AI: "Did the action succeed?"
|
|
5. If failed → Generate fix → Retry
|
|
```
|
|
|
|
## Grounding Map Example
|
|
|
|
```json
|
|
{
|
|
"screenshot": "base64://...",
|
|
"elements": [
|
|
{ "id": 1, "type": "button", "text": "File", "bounds": [10, 5, 50, 25] },
|
|
{ "id": 2, "type": "button", "text": "Edit", "bounds": [60, 5, 100, 25] },
|
|
{ "id": 3, "type": "input", "placeholder": "Search...", "bounds": [150, 50, 400, 80] },
|
|
{ "id": 4, "type": "button", "text": "Save", "bounds": [300, 500, 380, 530] },
|
|
{ "id": 5, "type": "button", "text": "Cancel", "bounds": [400, 500, 480, 530] }
|
|
],
|
|
"ocr_text": [
|
|
{ "text": "Untitled - Notepad", "bounds": [100, 0, 300, 20] },
|
|
{ "text": "Hello World", "bounds": [50, 100, 200, 120] }
|
|
]
|
|
}
|
|
```
|
|
|
|
## AI Prompt with Vision
|
|
|
|
```
|
|
You are controlling a Windows computer. Here's what you see:
|
|
|
|
[SCREENSHOT: base64 image]
|
|
|
|
INTERACTIVE ELEMENTS:
|
|
[1] Button: "File" at (10, 5)
|
|
[2] Button: "Edit" at (60, 5)
|
|
[3] Input: "Search..." at (150, 50)
|
|
[4] Button: "Save" at (300, 500)
|
|
[5] Button: "Cancel" at (400, 500)
|
|
|
|
VISIBLE TEXT:
|
|
- "Untitled - Notepad" (title bar)
|
|
- "Hello World" (document content)
|
|
|
|
USER REQUEST: "Save this file as hello.txt"
|
|
|
|
YOUR ACTION:
|
|
- To click an element, say: CLICK [4]
|
|
- To type, say: TYPE "hello.txt"
|
|
- To press key, say: KEY Enter
|
|
```
|
|
|
|
---
|
|
|
|
# 5. Vibe Server Management
|
|
|
|
## Purpose
|
|
**"Vibe Server Management"** = Server tasks for people who don't know servers.
|
|
|
|
No SSH commands to memorize. Just tell Goose what you want in plain English.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌────────────────────────────────────────────────────────────────┐
|
|
│ VIBE SERVER MANAGEMENT │
|
|
├────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ User: "My website is down, fix it" │
|
|
│ ↓ │
|
|
│ ┌──────────────────────────────────────────────────────────┐ │
|
|
│ │ IQ Exchange Translation Layer │ │
|
|
│ │ → SERVER action detected │ │
|
|
│ │ → Route to Server Management module │ │
|
|
│ └──────────────────────────────────────────────────────────┘ │
|
|
│ ↓ │
|
|
│ ┌──────────────────────────────────────────────────────────┐ │
|
|
│ │ SERVER DIAGNOSIS FLOW │ │
|
|
│ │ │ │
|
|
│ │ 1. Check if we have SSH credentials │ │
|
|
│ │ → If not: "I need SSH access. Enter credentials?" │ │
|
|
│ │ │ │
|
|
│ │ 2. Connect to server │ │
|
|
│ │ → ssh root@86.105.224.125 │ │
|
|
│ │ │ │
|
|
│ │ 3. Diagnose the issue │ │
|
|
│ │ → systemctl status nginx │ │
|
|
│ │ → pm2 status │ │
|
|
│ │ → tail -n 50 /var/log/nginx/error.log │ │
|
|
│ │ │ │
|
|
│ │ 4. Fix the issue │ │
|
|
│ │ → systemctl restart nginx │ │
|
|
│ │ → pm2 restart all │ │
|
|
│ │ │ │
|
|
│ │ 5. Verify fix worked │ │
|
|
│ │ → curl http://localhost │ │
|
|
│ │ → Report back to user │ │
|
|
│ └──────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
└────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## Noob-Friendly Commands (What User Says → What Goose Does)
|
|
|
|
| User Says | Goose Does |
|
|
|-----------|------------|
|
|
| "Is my server running?" | `ssh` → `systemctl status nginx` → parse output |
|
|
| "Restart my website" | `pm2 restart all` or `systemctl restart nginx` |
|
|
| "Check the logs" | `tail -n 100 /var/log/nginx/error.log` |
|
|
| "Deploy my latest code" | `git pull` → `npm install` → `npm run build` → `pm2 restart` |
|
|
| "My server is slow" | Check CPU/memory, identify resource hogs |
|
|
| "Set up SSL" | `certbot --nginx -d domain.com` |
|
|
| "Add a new website" | Create nginx config, set up PM2 process |
|
|
| "Backup my database" | `pg_dump` or `mysqldump` |
|
|
| "My disk is full" | `df -h`, find large files, clean up safely |
|
|
|
|
## Server Panel UI
|
|
|
|
```
|
|
┌────────────────────────────────────────────────────────────────┐
|
|
│ 🔗 SERVER MANAGEMENT [X] │
|
|
├────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ ┌───────────────────────────────────────────────────────────┐│
|
|
│ │ CONNECTIONS [+ Add New] ││
|
|
│ │ ───────────────────────────────────────────────────────── ││
|
|
│ │ ● Production Server (86.105.224.125) [Connect] ││
|
|
│ │ ○ Staging Server (192.168.1.100) [Connect] ││
|
|
│ └───────────────────────────────────────────────────────────┘│
|
|
│ │
|
|
│ ┌───────────────────────────────────────────────────────────┐│
|
|
│ │ QUICK ACTIONS ││
|
|
│ │ ───────────────────────────────────────────────────────── ││
|
|
│ │ [🔄 Restart Website] [📊 Check Status] [📋 View Logs] ││
|
|
│ │ [🚀 Deploy Latest] [💾 Backup DB] [🔒 SSL Setup] ││
|
|
│ └───────────────────────────────────────────────────────────┘│
|
|
│ │
|
|
│ ┌───────────────────────────────────────────────────────────┐│
|
|
│ │ LIVE TERMINAL ││
|
|
│ │ ───────────────────────────────────────────────────────── ││
|
|
│ │ root@server:~# systemctl status nginx ││
|
|
│ │ ● nginx.service - A high performance web server ││
|
|
│ │ Loaded: loaded (/lib/systemd/system/nginx.service) ││
|
|
│ │ Active: active (running) since Mon 2025-12-16 10:00 ││
|
|
│ │ ││
|
|
│ │ $> _ ││
|
|
│ └───────────────────────────────────────────────────────────┘│
|
|
│ │
|
|
│ ┌───────────────────────────────────────────────────────────┐│
|
|
│ │ 💬 ASK GOOSE (natural language) ││
|
|
│ │ ───────────────────────────────────────────────────────── ││
|
|
│ │ "Why is my website slow?" [Ask Goose] ││
|
|
│ └───────────────────────────────────────────────────────────┘│
|
|
│ │
|
|
└────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## SSH Service Implementation
|
|
|
|
```javascript
|
|
// services/SSHService.js
|
|
const { Client } = require('ssh2');
|
|
|
|
class SSHService {
|
|
constructor() {
|
|
this.connections = new Map();
|
|
}
|
|
|
|
async connect(name, config) {
|
|
const conn = new Client();
|
|
return new Promise((resolve, reject) => {
|
|
conn.on('ready', () => {
|
|
this.connections.set(name, conn);
|
|
resolve({ success: true });
|
|
});
|
|
conn.on('error', reject);
|
|
conn.connect(config);
|
|
});
|
|
}
|
|
|
|
async exec(name, command) {
|
|
const conn = this.connections.get(name);
|
|
if (!conn) throw new Error('Not connected');
|
|
|
|
return new Promise((resolve, reject) => {
|
|
conn.exec(command, (err, stream) => {
|
|
if (err) reject(err);
|
|
let output = '';
|
|
stream.on('data', (data) => output += data);
|
|
stream.on('close', () => resolve(output));
|
|
});
|
|
});
|
|
}
|
|
|
|
// High-level helper methods
|
|
async checkWebsiteStatus(name) {
|
|
const nginx = await this.exec(name, 'systemctl is-active nginx');
|
|
const pm2 = await this.exec(name, 'pm2 jlist');
|
|
return { nginx: nginx.trim(), pm2: JSON.parse(pm2) };
|
|
}
|
|
|
|
async restartWebsite(name) {
|
|
await this.exec(name, 'pm2 restart all');
|
|
await this.exec(name, 'systemctl reload nginx');
|
|
return { success: true };
|
|
}
|
|
|
|
async getLogs(name, lines = 50) {
|
|
return this.exec(name, `tail -n ${lines} /var/log/nginx/error.log`);
|
|
}
|
|
|
|
async deploy(name, path = '/var/www/app') {
|
|
await this.exec(name, `cd ${path} && git pull`);
|
|
await this.exec(name, `cd ${path} && npm install`);
|
|
await this.exec(name, `cd ${path} && npm run build`);
|
|
await this.exec(name, 'pm2 restart all');
|
|
return { success: true };
|
|
}
|
|
}
|
|
```
|
|
|
|
## Saved Server Configurations
|
|
|
|
```json
|
|
// .opencode/servers.json
|
|
{
|
|
"production": {
|
|
"name": "Production Server",
|
|
"host": "86.105.224.125",
|
|
"port": 22,
|
|
"username": "root",
|
|
"authType": "password",
|
|
"webRoot": "/var/www/myapp",
|
|
"pm2App": "myapp"
|
|
},
|
|
"staging": {
|
|
"name": "Staging Server",
|
|
"host": "192.168.1.100",
|
|
"port": 22,
|
|
"username": "deploy",
|
|
"authType": "key",
|
|
"keyPath": "~/.ssh/id_rsa",
|
|
"webRoot": "/var/www/staging"
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
# Summary: All Five Subsystems
|
|
|
|
| Subsystem | Purpose | Key Tech |
|
|
|-----------|---------|----------|
|
|
| **1. Playwright** | Browser automation | Playwright + persistent sessions |
|
|
| **2. Browser Use** | Built-in browser panel | Electron BrowserView |
|
|
| **3. Computer Use** | Desktop automation | input.ps1 + UIAutomation |
|
|
| **4. Full Vision** | See and verify | OCR + Screenshot + Grounding |
|
|
| **5. Vibe Server** | Noob-friendly server ops | SSH2 + high-level helpers |
|
|
|
|
All subsystems connect through **IQ Exchange** which:
|
|
- Detects task type from natural language
|
|
- Routes to appropriate subsystem
|
|
- Executes with self-healing loop
|
|
- Verifies success using vision
|
|
- Retries until success or max attempts
|
|
|
|
---
|
|
|
|
*This document provides implementation-ready specifications for each core subsystem.*
|